2024-04-28T12:30:51.163 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-28T12:30:51.164 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} --name rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890 --owner scheduled_rishabh@teuthology --verbose -- /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/orig.config.yaml 2024-04-28T12:30:51.220 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-28T12:30:51.314 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} email: null first_in_suite: false job_id: '7676890' kernel: client: sha1: distro flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMIgahWvWS44IS4b36QLV2QKG/hhVZLE+IK99bTQHlWdD6dTNYCzOaT6U1VxDQZ/pgyYoLGY8ZRRR1T/bVCjObw= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIAnM+YPplhW31N9E5+qTneFES7GFimTewRdhA/tPH8YZtEhWNilS+lv2ot4eO5COxAZ/K5HCOsX+3PPR/ZdnUU= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK1BN7AJyX3cYGmZJiBPTv1CHSvO8KDjNvOpzKLBm1uX67XwBql0ZibtywniMbW0yxUiWjAlJszo/XP5rh0tuic= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-28_11:41:23 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-28T12:30:51.314 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-28T12:30:51.315 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-28T12:30:51.315 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-28T12:30:51.316 INFO:teuthology.task.internal:Checking packages... 2024-04-28T12:30:51.340 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-28T12:30:51.340 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-28T12:30:51.340 INFO:teuthology.packaging:ref: None 2024-04-28T12:30:51.340 INFO:teuthology.packaging:tag: None 2024-04-28T12:30:51.340 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-28T12:30:51.340 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:30:51.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-28T12:30:51.553 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-28T12:30:51.555 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-28T12:30:51.562 INFO:teuthology.task.internal:no buildpackages task found 2024-04-28T12:30:51.563 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-28T12:30:51.615 INFO:teuthology.task.internal:Saving configuration 2024-04-28T12:30:51.636 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-28T12:30:51.647 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T12:30:51.672 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:25:11.300093', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMIgahWvWS44IS4b36QLV2QKG/hhVZLE+IK99bTQHlWdD6dTNYCzOaT6U1VxDQZ/pgyYoLGY8ZRRR1T/bVCjObw='} 2024-04-28T12:30:51.694 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:25:11.301400', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIAnM+YPplhW31N9E5+qTneFES7GFimTewRdhA/tPH8YZtEhWNilS+lv2ot4eO5COxAZ/K5HCOsX+3PPR/ZdnUU='} 2024-04-28T12:30:51.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:25:11.302753', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK1BN7AJyX3cYGmZJiBPTv1CHSvO8KDjNvOpzKLBm1uX67XwBql0ZibtywniMbW0yxUiWjAlJszo/XP5rh0tuic='} 2024-04-28T12:30:51.714 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-28T12:30:51.724 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-28T12:30:51.724 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-28T12:30:51.724 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-28T12:30:51.724 INFO:teuthology.run_tasks:Running task console_log... 2024-04-28T12:30:51.841 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5c8087e160>, signals=[15]) 2024-04-28T12:30:51.841 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-28T12:30:51.850 INFO:teuthology.task.internal:Opening connections... 2024-04-28T12:30:51.850 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-04-28T12:30:51.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:30:51.934 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-04-28T12:30:51.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:30:52.008 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-04-28T12:30:52.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:30:52.083 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-28T12:30:52.091 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-04-28T12:30:52.111 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-04-28T12:30:52.111 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T12:30:52.170 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-04-28T12:30:52.171 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-04-28T12:30:52.171 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T12:30:52.171 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-04-28T12:30:52.171 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T12:30:52.171 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T12:30:52.171 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T12:30:52.172 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-04-28T12:30:52.196 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-04-28T12:30:52.214 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-04-28T12:30:52.214 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-04-28T12:30:52.271 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-04-28T12:30:52.271 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-04-28T12:30:52.271 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-04-28T12:30:52.271 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-04-28T12:30:52.271 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-04-28T12:30:52.271 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T12:30:52.272 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T12:30:52.272 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-04-28T12:30:52.296 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-04-28T12:30:52.314 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-04-28T12:30:52.314 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-04-28T12:30:52.372 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-04-28T12:30:52.373 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T12:30:52.373 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-04-28T12:30:52.373 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T12:30:52.373 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T12:30:52.373 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T12:30:52.373 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-04-28T12:30:52.404 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-28T12:30:52.415 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-28T12:30:52.422 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-28T12:30:52.422 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-04-28T12:30:52.425 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-04-28T12:30:52.427 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-04-28T12:30:52.444 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-28T12:30:52.451 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-28T12:30:52.451 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-04-28T12:30:52.483 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-04-28T12:30:52.485 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-04-28T12:30:52.506 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-28T12:30:52.588 INFO:teuthology.run_tasks:Running task kernel... 2024-04-28T12:30:52.603 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-28T12:30:52.603 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-28T12:30:52.603 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-28T12:30:52.604 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T12:30:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:30:52.622 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-04-28T12:30:52.680 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-437.el9.x86_64 2024-04-28T12:30:52.680 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-437.el9.x86_64 2024-04-28T12:30:52.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-04-28T12:31:03.502 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 894 kB/s | 8.0 MB 00:09 2024-04-28T12:31:05.646 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-28T12:31:10.306 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.2 MB 00:00 2024-04-28T12:31:11.763 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-04-28T12:31:13.522 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-28T12:31:18.747 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-28T12:31:20.662 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-28T12:31:20.662 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-28T12:31:20.749 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-28T12:31:20.750 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-04-28T12:31:20.750 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-28T12:31:20.885 DEBUG:teuthology.orchestra.run.smithi053:> echo no | sudo yum reinstall kernel || true 2024-04-28T12:31:21.405 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:03 ago on Sun 28 Apr 2024 12:31:18 PM UTC. 2024-04-28T12:31:21.528 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout:Reinstalling: 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-28T12:31:21.549 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-28T12:31:21.550 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:31:21.550 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 255 k 2024-04-28T12:31:21.550 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 0 2024-04-28T12:31:21.550 INFO:teuthology.orchestra.run.smithi053.stderr:Operation aborted. 2024-04-28T12:31:21.609 INFO:teuthology.orchestra.run.smithi053.stdout:Is this ok [y/N]: 2024-04-28T12:31:21.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum reinstall -y kernel || true 2024-04-28T12:31:22.134 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:04 ago on Sun 28 Apr 2024 12:31:18 PM UTC. 2024-04-28T12:31:22.255 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout:Reinstalling: 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 255 k 2024-04-28T12:31:22.256 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 0 2024-04-28T12:31:22.257 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-04-28T12:31:22.541 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.9 MB/s | 255 kB 00:00 2024-04-28T12:31:22.541 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:31:22.542 INFO:teuthology.orchestra.run.smithi053.stdout:Total 895 kB/s | 255 kB 00:00 2024-04-28T12:31:22.558 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-04-28T12:31:22.761 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-04-28T12:31:22.762 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-04-28T12:31:22.767 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-04-28T12:31:22.767 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-04-28T12:31:22.922 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-04-28T12:31:22.992 INFO:teuthology.orchestra.run.smithi053.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-28T12:31:22.992 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-28T12:31:23.694 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-28T12:31:23.694 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-28T12:31:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-28T12:31:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:31:24.392 INFO:teuthology.orchestra.run.smithi053.stdout:Reinstalled: 2024-04-28T12:31:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-28T12:31:24.392 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:31:24.392 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-28T12:31:24.631 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-04-28T12:31:24.680 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-28T12:31:24.680 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-28T12:31:24.680 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-28T12:31:24.681 DEBUG:teuthology.parallel:result is None 2024-04-28T12:31:24.681 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-28T12:31:24.689 INFO:teuthology.task.internal:Creating test directory... 2024-04-28T12:31:24.689 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T12:31:24.691 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T12:31:24.694 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T12:31:24.712 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-28T12:31:24.719 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-28T12:31:24.727 INFO:teuthology.task.internal:Creating archive directory... 2024-04-28T12:31:24.727 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T12:31:24.746 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T12:31:24.752 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T12:31:24.874 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-28T12:31:24.882 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-28T12:31:24.882 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T12:31:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:31:24.900 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T12:31:24.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:31:24.917 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T12:31:24.920 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T12:31:24.942 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T12:31:24.970 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:31:24.970 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:31:24.984 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:31:24.989 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:31:25.007 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:31:25.009 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:31:25.010 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-28T12:31:25.018 INFO:teuthology.task.internal:Configuring sudo... 2024-04-28T12:31:25.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T12:31:25.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T12:31:25.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T12:31:25.078 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-28T12:31:25.091 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-28T12:31:25.091 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T12:31:25.094 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T12:31:25.119 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T12:31:25.135 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:31:25.191 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:31:25.242 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:31:25.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:31:25.363 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:31:25.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T12:31:25.427 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:31:25.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:31:25.516 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:31:25.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:31:25.639 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:31:25.639 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T12:31:25.703 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:31:25.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:31:25.801 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:31:25.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:31:25.926 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:31:25.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T12:31:25.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-04-28T12:31:25.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-04-28T12:31:25.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-04-28T12:31:26.032 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T12:31:26.038 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T12:31:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T12:31:26.570 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-28T12:31:26.579 INFO:teuthology.task.internal:Starting timer... 2024-04-28T12:31:26.579 INFO:teuthology.run_tasks:Running task pcp... 2024-04-28T12:31:26.591 INFO:teuthology.run_tasks:Running task selinux... 2024-04-28T12:31:26.600 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-28T12:31:26.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-04-28T12:31:26.665 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-04-28T12:31:26.667 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-04-28T12:31:26.723 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-04-28T12:31:26.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-04-28T12:31:26.784 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-04-28T12:31:26.786 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-28T12:31:26.786 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-04-28T12:31:26.810 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-04-28T12:31:26.811 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-04-28T12:31:26.839 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-04-28T12:31:26.839 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-04-28T12:31:26.868 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-04-28T12:31:26.868 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-04-28T12:31:26.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T12:31:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:31:26.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T12:31:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:31:26.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T12:31:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:31:26.947 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-28T12:31:26.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-04-28T12:31:26.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-04-28T12:31:26.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-04-28T12:31:27.028 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-28T12:31:27.038 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-28T12:31:27.057 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-28T12:31:27.059 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi077.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-28T12:35:17.287 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-04-28T12:35:17.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-28T12:35:17.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:35:17.369 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-04-28T12:35:17.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-28T12:35:17.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-28T12:35:17.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:35:17.533 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-04-28T12:35:17.619 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-28T12:35:17.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-04-28T12:35:17.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:35:17.693 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-04-28T12:35:17.773 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-04-28T12:35:17.774 INFO:teuthology.run_tasks:Running task clock... 2024-04-28T12:35:17.784 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-28T12:35:17.784 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T12:35:17.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T12:35:17.787 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T12:35:17.787 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T12:35:17.790 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T12:35:17.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T12:35:17.816 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T12:35:17.820 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T12:35:17.831 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T12:35:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T12:35:17.843 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T12:35:17.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-04-28T12:35:17.856 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T12:35:17.859 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-04-28T12:35:17.867 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-04-28T12:35:17.871 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-04-28T12:35:17.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-04-28T12:35:17.881 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T12:35:17.885 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T12:35:17.890 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-04-28T12:35:17.895 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T12:35:17.898 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T12:35:17.903 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T12:35:17.916 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T12:35:17.951 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-28T12:35:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-28T12:35:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T12:35:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-28T12:35:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.956 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T12:35:17.956 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-28T12:35:17.957 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.957 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.957 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.957 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.968 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-04-28T12:35:17.971 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T12:35:17.971 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-04-28T12:35:17.971 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.971 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.972 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.972 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:35:17.972 INFO:teuthology.run_tasks:Running task install... 2024-04-28T12:35:17.981 DEBUG:teuthology.task.install:project ceph 2024-04-28T12:35:17.981 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-28T12:35:17.981 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-28T12:35:17.981 INFO:teuthology.task.install:Using flavor: default 2024-04-28T12:35:17.987 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-28T12:35:17.987 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-28T12:35:17.988 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T12:35:17.988 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:17.990 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T12:35:17.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:17.991 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T12:35:17.991 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:18.142 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T12:35:18.142 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T12:35:18.146 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T12:35:18.146 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T12:35:18.283 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T12:35:18.283 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T12:35:18.426 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T12:35:18.426 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:35:18.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T12:35:18.452 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T12:35:18.452 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T12:35:18.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T12:35:18.563 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T12:35:18.563 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:35:18.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T12:35:18.589 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T12:35:18.590 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T12:35:18.591 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T12:35:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-04-28T12:35:18.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T12:35:18.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-04-28T12:35:18.671 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T12:35:18.671 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:35:18.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T12:35:18.699 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T12:35:18.699 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T12:35:18.728 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T12:35:18.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T12:35:18.791 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-04-28T12:35:18.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-04-28T12:35:18.844 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T12:35:18.881 INFO:teuthology.orchestra.run.smithi077.stdout:48 files removed 2024-04-28T12:35:18.908 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T12:35:18.911 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-04-28T12:35:18.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-04-28T12:35:19.016 INFO:teuthology.orchestra.run.smithi191.stdout:48 files removed 2024-04-28T12:35:19.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T12:35:19.149 INFO:teuthology.orchestra.run.smithi053.stdout:48 files removed 2024-04-28T12:35:19.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T12:35:20.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 88 kB/s | 84 kB 00:00 2024-04-28T12:35:20.206 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 111 kB/s | 84 kB 00:00 2024-04-28T12:35:20.439 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 74 kB/s | 84 kB 00:01 2024-04-28T12:35:20.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 15 kB/s | 12 kB 00:00 2024-04-28T12:35:21.223 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 12 kB/s | 12 kB 00:00 2024-04-28T12:35:21.298 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 14 kB/s | 12 kB 00:00 2024-04-28T12:35:21.969 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 1.9 kB/s | 1.9 kB 00:00 2024-04-28T12:35:22.224 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-04-28T12:35:22.282 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-04-28T12:35:22.890 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-28T12:35:23.978 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-28T12:35:24.617 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-04-28T12:35:25.551 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-04-28T12:35:27.516 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-28T12:35:29.042 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-28T12:35:29.662 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 19 MB 00:04 2024-04-28T12:35:30.668 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-28T12:35:32.270 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-28T12:35:32.816 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.2 MB 00:01 2024-04-28T12:35:33.863 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-28T12:35:34.475 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-28T12:35:35.422 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-28T12:35:36.110 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-28T12:35:37.603 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-28T12:35:37.659 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-04-28T12:35:39.803 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:35:39.804 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:35:39.815 INFO:teuthology.orchestra.run.smithi053.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T12:35:39.816 INFO:teuthology.orchestra.run.smithi053.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T12:35:39.817 INFO:teuthology.orchestra.run.smithi053.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T12:35:39.818 INFO:teuthology.orchestra.run.smithi053.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T12:35:39.819 INFO:teuthology.orchestra.run.smithi053.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T12:35:39.822 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T12:35:39.822 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T12:35:39.890 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-28T12:35:39.899 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-28T12:35:39.899 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-04-28T12:35:39.899 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-28T12:35:39.899 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-04-28T12:35:39.899 INFO:teuthology.orchestra.run.smithi053.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T12:35:39.899 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T12:35:39.899 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T12:35:39.900 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T12:35:39.901 INFO:teuthology.orchestra.run.smithi053.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T12:35:39.902 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T12:35:39.903 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T12:35:39.904 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T12:35:39.905 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T12:35:39.906 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T12:35:39.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T12:35:39.908 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T12:35:39.909 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T12:35:39.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T12:35:39.911 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout:Install 148 Packages 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 4 Packages 2024-04-28T12:35:39.912 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:35:39.913 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 222 M 2024-04-28T12:35:39.913 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-04-28T12:35:40.896 INFO:teuthology.orchestra.run.smithi053.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-28T12:35:41.110 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-04-28T12:35:41.971 INFO:teuthology.orchestra.run.smithi053.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 827 kB/s | 889 kB 00:01 2024-04-28T12:35:42.172 INFO:teuthology.orchestra.run.smithi053.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 726 kB/s | 145 kB 00:00 2024-04-28T12:35:42.806 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 7.8 kB/s | 1.7 kB 00:00 2024-04-28T12:35:43.220 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:35:43.220 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:35:43.231 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T12:35:43.232 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T12:35:43.233 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T12:35:43.234 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T12:35:43.235 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T12:35:43.238 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T12:35:43.238 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T12:35:43.298 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-04-28T12:35:43.308 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-04-28T12:35:43.308 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-04-28T12:35:43.308 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T12:35:43.309 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T12:35:43.310 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T12:35:43.311 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T12:35:43.311 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T12:35:43.311 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T12:35:43.311 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T12:35:43.311 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T12:35:43.311 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T12:35:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T12:35:43.313 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T12:35:43.314 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T12:35:43.315 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T12:35:43.316 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T12:35:43.317 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T12:35:43.318 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T12:35:43.319 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout:Install 148 Packages 2024-04-28T12:35:43.320 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 4 Packages 2024-04-28T12:35:43.321 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:35:43.321 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 222 M 2024-04-28T12:35:43.321 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-04-28T12:35:43.638 INFO:teuthology.orchestra.run.smithi053.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-28T12:35:44.213 INFO:teuthology.orchestra.run.smithi053.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-04-28T12:35:44.405 INFO:teuthology.orchestra.run.smithi053.stdout:(6/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 1.1 MB/s | 2.4 MB 00:02 2024-04-28T12:35:44.661 INFO:teuthology.orchestra.run.smithi191.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-28T12:35:44.895 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:35:44.896 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:35:44.907 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T12:35:44.908 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T12:35:44.908 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T12:35:44.909 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T12:35:44.911 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T12:35:44.913 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T12:35:44.914 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T12:35:44.975 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T12:35:44.996 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T12:35:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T12:35:44.998 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-04-28T12:35:44.999 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T12:35:45.000 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T12:35:45.001 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T12:35:45.002 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T12:35:45.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T12:35:45.004 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T12:35:45.005 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T12:35:45.006 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-04-28T12:35:45.007 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-28T12:35:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:Install 148 Packages 2024-04-28T12:35:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 4 Packages 2024-04-28T12:35:45.008 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:35:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 222 M 2024-04-28T12:35:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-04-28T12:35:45.553 INFO:teuthology.orchestra.run.smithi191.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 998 kB/s | 889 kB 00:00 2024-04-28T12:35:45.720 INFO:teuthology.orchestra.run.smithi191.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 871 kB/s | 145 kB 00:00 2024-04-28T12:35:45.781 INFO:teuthology.orchestra.run.smithi053.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-04-28T12:35:45.974 INFO:teuthology.orchestra.run.smithi077.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-28T12:35:46.921 INFO:teuthology.orchestra.run.smithi191.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 2.4 MB 00:01 2024-04-28T12:35:47.125 INFO:teuthology.orchestra.run.smithi077.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 773 kB/s | 889 kB 00:01 2024-04-28T12:35:47.325 INFO:teuthology.orchestra.run.smithi077.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 726 kB/s | 145 kB 00:00 2024-04-28T12:35:47.637 INFO:teuthology.orchestra.run.smithi191.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-04-28T12:35:48.900 INFO:teuthology.orchestra.run.smithi077.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 1.5 MB/s | 2.4 MB 00:01 2024-04-28T12:35:49.439 INFO:teuthology.orchestra.run.smithi053.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 2.9 MB/s | 11 MB 00:03 2024-04-28T12:35:49.522 INFO:teuthology.orchestra.run.smithi053.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 303 kB/s | 25 kB 00:00 2024-04-28T12:35:49.762 INFO:teuthology.orchestra.run.smithi191.stdout:(6/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-04-28T12:35:49.809 INFO:teuthology.orchestra.run.smithi077.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 1.7 MB/s | 1.5 MB 00:00 2024-04-28T12:35:50.267 INFO:teuthology.orchestra.run.smithi077.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.1 MB/s | 5.4 MB 00:04 2024-04-28T12:35:52.621 INFO:teuthology.orchestra.run.smithi191.stdout:(7/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 663 kB/s | 5.4 MB 00:08 2024-04-28T12:35:53.234 INFO:teuthology.orchestra.run.smithi077.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 1.4 MB/s | 4.7 MB 00:03 2024-04-28T12:35:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 1.9 MB/s | 17 MB 00:08 2024-04-28T12:35:53.330 INFO:teuthology.orchestra.run.smithi053.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.4 MB/s | 225 kB 00:00 2024-04-28T12:35:53.422 INFO:teuthology.orchestra.run.smithi053.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 294 kB/s | 27 kB 00:00 2024-04-28T12:35:53.515 INFO:teuthology.orchestra.run.smithi053.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 347 kB/s | 32 kB 00:00 2024-04-28T12:35:54.256 INFO:teuthology.orchestra.run.smithi053.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.0 MB/s | 747 kB 00:00 2024-04-28T12:35:54.348 INFO:teuthology.orchestra.run.smithi053.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.7 MB/s | 163 kB 00:00 2024-04-28T12:35:54.457 INFO:teuthology.orchestra.run.smithi053.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.1 MB/s | 127 kB 00:00 2024-04-28T12:35:54.716 INFO:teuthology.orchestra.run.smithi053.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 1.9 MB/s | 501 kB 00:00 2024-04-28T12:35:56.266 INFO:teuthology.orchestra.run.smithi053.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 3.5 MB/s | 5.4 MB 00:01 2024-04-28T12:35:56.341 INFO:teuthology.orchestra.run.smithi053.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 604 kB/s | 45 kB 00:00 2024-04-28T12:35:56.450 INFO:teuthology.orchestra.run.smithi053.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.2 MB/s | 135 kB 00:00 2024-04-28T12:35:56.534 INFO:teuthology.orchestra.run.smithi053.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.9 MB/s | 161 kB 00:00 2024-04-28T12:35:56.667 INFO:teuthology.orchestra.run.smithi053.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 2.4 MB/s | 321 kB 00:00 2024-04-28T12:35:56.792 INFO:teuthology.orchestra.run.smithi053.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 2.3 MB/s | 298 kB 00:00 2024-04-28T12:35:56.867 INFO:teuthology.orchestra.run.smithi053.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.3 MB/s | 99 kB 00:00 2024-04-28T12:35:56.951 INFO:teuthology.orchestra.run.smithi053.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-28T12:35:57.712 INFO:teuthology.orchestra.run.smithi191.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 1.6 MB/s | 21 MB 00:13 2024-04-28T12:35:57.852 INFO:teuthology.orchestra.run.smithi053.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 3.4 MB/s | 3.1 MB 00:00 2024-04-28T12:35:57.920 INFO:teuthology.orchestra.run.smithi191.stdout:(9/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 2.1 MB/s | 17 MB 00:08 2024-04-28T12:35:57.935 INFO:teuthology.orchestra.run.smithi053.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-28T12:35:57.945 INFO:teuthology.orchestra.run.smithi191.stdout:(10/152): ceph-selinux-19.0.0-3306.gbc612ffb.el 108 kB/s | 25 kB 00:00 2024-04-28T12:35:58.010 INFO:teuthology.orchestra.run.smithi053.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 456 kB/s | 34 kB 00:00 2024-04-28T12:35:58.094 INFO:teuthology.orchestra.run.smithi053.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.7 MB/s | 142 kB 00:00 2024-04-28T12:35:58.104 INFO:teuthology.orchestra.run.smithi191.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 1.4 MB/s | 225 kB 00:00 2024-04-28T12:35:58.187 INFO:teuthology.orchestra.run.smithi191.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 325 kB/s | 27 kB 00:00 2024-04-28T12:35:58.192 INFO:teuthology.orchestra.run.smithi077.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 1.7 MB/s | 21 MB 00:12 2024-04-28T12:35:58.279 INFO:teuthology.orchestra.run.smithi191.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 346 kB/s | 32 kB 00:00 2024-04-28T12:35:58.309 INFO:teuthology.orchestra.run.smithi077.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 215 kB/s | 25 kB 00:00 2024-04-28T12:35:58.721 INFO:teuthology.orchestra.run.smithi191.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.7 MB/s | 747 kB 00:00 2024-04-28T12:35:58.888 INFO:teuthology.orchestra.run.smithi191.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 981 kB/s | 163 kB 00:00 2024-04-28T12:35:58.980 INFO:teuthology.orchestra.run.smithi191.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.4 MB/s | 127 kB 00:00 2024-04-28T12:35:59.010 INFO:teuthology.orchestra.run.smithi053.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 3.9 MB/s | 3.5 MB 00:00 2024-04-28T12:35:59.264 INFO:teuthology.orchestra.run.smithi191.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 1.7 MB/s | 501 kB 00:00 2024-04-28T12:36:00.953 INFO:teuthology.orchestra.run.smithi053.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 3.8 MB/s | 7.4 MB 00:01 2024-04-28T12:36:01.053 INFO:teuthology.orchestra.run.smithi053.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.4 MB/s | 249 kB 00:00 2024-04-28T12:36:01.145 INFO:teuthology.orchestra.run.smithi053.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 538 kB/s | 49 kB 00:00 2024-04-28T12:36:01.237 INFO:teuthology.orchestra.run.smithi053.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 180 kB/s | 16 kB 00:00 2024-04-28T12:36:01.337 INFO:teuthology.orchestra.run.smithi053.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.7 MB/s | 273 kB 00:00 2024-04-28T12:36:01.535 INFO:teuthology.orchestra.run.smithi077.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 1.5 MB/s | 17 MB 00:11 2024-04-28T12:36:01.604 INFO:teuthology.orchestra.run.smithi053.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 2.8 MB/s | 762 kB 00:00 2024-04-28T12:36:01.696 INFO:teuthology.orchestra.run.smithi053.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 199 kB/s | 18 kB 00:00 2024-04-28T12:36:01.755 INFO:teuthology.orchestra.run.smithi053.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 80 kB 00:00 2024-04-28T12:36:01.827 INFO:teuthology.orchestra.run.smithi077.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 773 kB/s | 225 kB 00:00 2024-04-28T12:36:01.855 INFO:teuthology.orchestra.run.smithi053.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.9 MB/s | 803 kB 00:00 2024-04-28T12:36:01.922 INFO:teuthology.orchestra.run.smithi053.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-28T12:36:01.960 INFO:teuthology.orchestra.run.smithi077.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 203 kB/s | 27 kB 00:00 2024-04-28T12:36:01.997 INFO:teuthology.orchestra.run.smithi053.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-28T12:36:02.061 INFO:teuthology.orchestra.run.smithi077.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 317 kB/s | 32 kB 00:00 2024-04-28T12:36:02.072 INFO:teuthology.orchestra.run.smithi191.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 1.9 MB/s | 5.4 MB 00:02 2024-04-28T12:36:02.114 INFO:teuthology.orchestra.run.smithi053.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-28T12:36:02.172 INFO:teuthology.orchestra.run.smithi191.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 453 kB/s | 45 kB 00:00 2024-04-28T12:36:02.181 INFO:teuthology.orchestra.run.smithi053.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-28T12:36:02.248 INFO:teuthology.orchestra.run.smithi053.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-28T12:36:02.256 INFO:teuthology.orchestra.run.smithi191.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.6 MB/s | 135 kB 00:00 2024-04-28T12:36:02.315 INFO:teuthology.orchestra.run.smithi053.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-28T12:36:02.378 INFO:teuthology.orchestra.run.smithi077.stdout:(14/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 1.2 MB/s | 11 MB 00:09 2024-04-28T12:36:02.390 INFO:teuthology.orchestra.run.smithi053.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-28T12:36:02.398 INFO:teuthology.orchestra.run.smithi191.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.1 MB/s | 161 kB 00:00 2024-04-28T12:36:02.553 INFO:teuthology.orchestra.run.smithi077.stdout:(15/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.5 MB/s | 747 kB 00:00 2024-04-28T12:36:02.573 INFO:teuthology.orchestra.run.smithi191.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 1.8 MB/s | 321 kB 00:00 2024-04-28T12:36:02.578 INFO:teuthology.orchestra.run.smithi077.stdout:(16/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 816 kB/s | 163 kB 00:00 2024-04-28T12:36:02.703 INFO:teuthology.orchestra.run.smithi077.stdout:(17/152): librados-devel-19.0.0-3306.gbc612ffb. 845 kB/s | 127 kB 00:00 2024-04-28T12:36:02.740 INFO:teuthology.orchestra.run.smithi191.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 1.7 MB/s | 298 kB 00:00 2024-04-28T12:36:02.857 INFO:teuthology.orchestra.run.smithi191.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 849 kB/s | 99 kB 00:00 2024-04-28T12:36:02.958 INFO:teuthology.orchestra.run.smithi191.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 853 kB/s | 85 kB 00:00 2024-04-28T12:36:02.991 INFO:teuthology.orchestra.run.smithi053.stdout:(47/152): ceph-common-19.0.0-3306.gbc612ffb.el9 963 kB/s | 21 MB 00:22 2024-04-28T12:36:02.995 INFO:teuthology.orchestra.run.smithi077.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 1.2 MB/s | 501 kB 00:00 2024-04-28T12:36:03.121 INFO:teuthology.orchestra.run.smithi077.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 361 kB/s | 45 kB 00:00 2024-04-28T12:36:03.249 INFO:teuthology.orchestra.run.smithi053.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 802 kB/s | 689 kB 00:00 2024-04-28T12:36:03.338 INFO:teuthology.orchestra.run.smithi077.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 623 kB/s | 135 kB 00:00 2024-04-28T12:36:03.358 INFO:teuthology.orchestra.run.smithi053.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 989 kB/s | 107 kB 00:00 2024-04-28T12:36:03.474 INFO:teuthology.orchestra.run.smithi053.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 184 kB/s | 21 kB 00:00 2024-04-28T12:36:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 775 kB/s | 161 kB 00:00 2024-04-28T12:36:03.583 INFO:teuthology.orchestra.run.smithi053.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-28T12:36:03.692 INFO:teuthology.orchestra.run.smithi053.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-04-28T12:36:03.784 INFO:teuthology.orchestra.run.smithi053.stdout:(53/152): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-04-28T12:36:03.846 INFO:teuthology.orchestra.run.smithi077.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 1.0 MB/s | 321 kB 00:00 2024-04-28T12:36:03.892 INFO:teuthology.orchestra.run.smithi053.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-04-28T12:36:04.001 INFO:teuthology.orchestra.run.smithi053.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 103 kB/s | 11 kB 00:00 2024-04-28T12:36:04.118 INFO:teuthology.orchestra.run.smithi053.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-28T12:36:04.147 INFO:teuthology.orchestra.run.smithi077.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 993 kB/s | 298 kB 00:00 2024-04-28T12:36:04.168 INFO:teuthology.orchestra.run.smithi053.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.3 MB/s | 3.0 MB 00:00 2024-04-28T12:36:04.272 INFO:teuthology.orchestra.run.smithi077.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 794 kB/s | 99 kB 00:00 2024-04-28T12:36:04.312 INFO:teuthology.orchestra.run.smithi053.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 233 kB/s | 45 kB 00:00 2024-04-28T12:36:04.346 INFO:teuthology.orchestra.run.smithi053.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 3.6 MB/s | 662 kB 00:00 2024-04-28T12:36:04.358 INFO:teuthology.orchestra.run.smithi191.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 2.2 MB/s | 3.1 MB 00:01 2024-04-28T12:36:04.430 INFO:teuthology.orchestra.run.smithi053.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-28T12:36:04.455 INFO:teuthology.orchestra.run.smithi053.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-04-28T12:36:04.458 INFO:teuthology.orchestra.run.smithi191.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-28T12:36:04.522 INFO:teuthology.orchestra.run.smithi077.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 341 kB/s | 85 kB 00:00 2024-04-28T12:36:04.538 INFO:teuthology.orchestra.run.smithi053.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-04-28T12:36:04.542 INFO:teuthology.orchestra.run.smithi191.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 411 kB/s | 34 kB 00:00 2024-04-28T12:36:04.572 INFO:teuthology.orchestra.run.smithi053.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-28T12:36:04.642 INFO:teuthology.orchestra.run.smithi191.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.4 MB/s | 142 kB 00:00 2024-04-28T12:36:04.656 INFO:teuthology.orchestra.run.smithi053.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-28T12:36:04.690 INFO:teuthology.orchestra.run.smithi053.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 301 kB/s | 35 kB 00:00 2024-04-28T12:36:04.751 INFO:teuthology.orchestra.run.smithi191.stdout:(30/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 894 kB/s | 11 MB 00:12 2024-04-28T12:36:05.548 INFO:teuthology.orchestra.run.smithi077.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 1.9 MB/s | 5.4 MB 00:02 2024-04-28T12:36:05.665 INFO:teuthology.orchestra.run.smithi077.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 1.4 MB/s | 171 kB 00:00 2024-04-28T12:36:05.782 INFO:teuthology.orchestra.run.smithi077.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 293 kB/s | 34 kB 00:00 2024-04-28T12:36:05.899 INFO:teuthology.orchestra.run.smithi077.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.2 MB/s | 142 kB 00:00 2024-04-28T12:36:06.115 INFO:teuthology.orchestra.run.smithi053.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.2 MB/s | 4.7 MB 00:01 2024-04-28T12:36:06.274 INFO:teuthology.orchestra.run.smithi077.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 1.8 MB/s | 3.1 MB 00:01 2024-04-28T12:36:06.518 INFO:teuthology.orchestra.run.smithi191.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 1.9 MB/s | 3.5 MB 00:01 2024-04-28T12:36:06.660 INFO:teuthology.orchestra.run.smithi191.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 1.7 MB/s | 249 kB 00:00 2024-04-28T12:36:06.726 INFO:teuthology.orchestra.run.smithi191.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 741 kB/s | 49 kB 00:00 2024-04-28T12:36:06.810 INFO:teuthology.orchestra.run.smithi191.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 197 kB/s | 16 kB 00:00 2024-04-28T12:36:06.961 INFO:teuthology.orchestra.run.smithi191.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 1.8 MB/s | 273 kB 00:00 2024-04-28T12:36:07.183 INFO:teuthology.orchestra.run.smithi077.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 2.8 MB/s | 3.5 MB 00:01 2024-04-28T12:36:07.300 INFO:teuthology.orchestra.run.smithi077.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.1 MB/s | 249 kB 00:00 2024-04-28T12:36:07.327 INFO:teuthology.orchestra.run.smithi191.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 2.0 MB/s | 762 kB 00:00 2024-04-28T12:36:07.400 INFO:teuthology.orchestra.run.smithi077.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 493 kB/s | 49 kB 00:00 2024-04-28T12:36:07.411 INFO:teuthology.orchestra.run.smithi191.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 218 kB/s | 18 kB 00:00 2024-04-28T12:36:07.509 INFO:teuthology.orchestra.run.smithi077.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 152 kB/s | 16 kB 00:00 2024-04-28T12:36:07.520 INFO:teuthology.orchestra.run.smithi191.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 740 kB/s | 80 kB 00:00 2024-04-28T12:36:07.620 INFO:teuthology.orchestra.run.smithi191.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.9 MB/s | 803 kB 00:00 2024-04-28T12:36:07.662 INFO:teuthology.orchestra.run.smithi191.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-28T12:36:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 273 kB 00:00 2024-04-28T12:36:07.712 INFO:teuthology.orchestra.run.smithi191.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-28T12:36:07.779 INFO:teuthology.orchestra.run.smithi191.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-28T12:36:07.821 INFO:teuthology.orchestra.run.smithi191.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-28T12:36:07.863 INFO:teuthology.orchestra.run.smithi191.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-28T12:36:07.905 INFO:teuthology.orchestra.run.smithi191.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-28T12:36:07.947 INFO:teuthology.orchestra.run.smithi191.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.0 MB/s | 215 kB 00:00 2024-04-28T12:36:07.984 INFO:teuthology.orchestra.run.smithi077.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 2.5 MB/s | 762 kB 00:00 2024-04-28T12:36:08.042 INFO:teuthology.orchestra.run.smithi191.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 7.3 MB/s | 689 kB 00:00 2024-04-28T12:36:08.084 INFO:teuthology.orchestra.run.smithi077.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 183 kB/s | 18 kB 00:00 2024-04-28T12:36:08.156 INFO:teuthology.orchestra.run.smithi191.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 7.9 MB/s | 943 kB 00:00 2024-04-28T12:36:08.223 INFO:teuthology.orchestra.run.smithi191.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-28T12:36:08.274 INFO:teuthology.orchestra.run.smithi191.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-04-28T12:36:08.349 INFO:teuthology.orchestra.run.smithi191.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-04-28T12:36:08.381 INFO:teuthology.orchestra.run.smithi053.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.2 MB/s | 4.4 MB 00:03 2024-04-28T12:36:08.391 INFO:teuthology.orchestra.run.smithi191.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 782 kB/s | 33 kB 00:00 2024-04-28T12:36:08.423 INFO:teuthology.orchestra.run.smithi053.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 704 kB/s | 1.6 MB 00:02 2024-04-28T12:36:08.568 INFO:teuthology.orchestra.run.smithi077.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 3.2 MB/s | 7.4 MB 00:02 2024-04-28T12:36:08.591 INFO:teuthology.orchestra.run.smithi191.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-04-28T12:36:08.633 INFO:teuthology.orchestra.run.smithi191.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 438 kB/s | 18 kB 00:00 2024-04-28T12:36:08.667 INFO:teuthology.orchestra.run.smithi191.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 333 kB/s | 11 kB 00:00 2024-04-28T12:36:08.690 INFO:teuthology.orchestra.run.smithi053.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 500 kB/s | 133 kB 00:00 2024-04-28T12:36:08.734 INFO:teuthology.orchestra.run.smithi191.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-04-28T12:36:08.784 INFO:teuthology.orchestra.run.smithi191.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-04-28T12:36:08.860 INFO:teuthology.orchestra.run.smithi077.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 104 kB/s | 80 kB 00:00 2024-04-28T12:36:08.884 INFO:teuthology.orchestra.run.smithi191.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-28T12:36:08.976 INFO:teuthology.orchestra.run.smithi191.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-04-28T12:36:09.018 INFO:teuthology.orchestra.run.smithi191.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 611 kB/s | 25 kB 00:00 2024-04-28T12:36:09.077 INFO:teuthology.orchestra.run.smithi191.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-04-28T12:36:09.152 INFO:teuthology.orchestra.run.smithi191.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-28T12:36:09.219 INFO:teuthology.orchestra.run.smithi191.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-28T12:36:09.269 INFO:teuthology.orchestra.run.smithi191.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-28T12:36:09.512 INFO:teuthology.orchestra.run.smithi191.stdout:(65/152): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-04-28T12:36:09.632 INFO:teuthology.orchestra.run.smithi053.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 729 kB/s | 911 kB 00:01 2024-04-28T12:36:09.702 INFO:teuthology.orchestra.run.smithi077.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 232 kB/s | 195 kB 00:00 2024-04-28T12:36:09.737 INFO:teuthology.orchestra.run.smithi191.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-28T12:36:09.870 INFO:teuthology.orchestra.run.smithi191.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-28T12:36:09.979 INFO:teuthology.orchestra.run.smithi191.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-28T12:36:10.038 INFO:teuthology.orchestra.run.smithi191.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-28T12:36:10.127 INFO:teuthology.orchestra.run.smithi077.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 515 kB/s | 803 kB 00:01 2024-04-28T12:36:10.297 INFO:teuthology.orchestra.run.smithi191.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-28T12:36:10.422 INFO:teuthology.orchestra.run.smithi191.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-28T12:36:10.480 INFO:teuthology.orchestra.run.smithi191.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-28T12:36:10.633 INFO:teuthology.orchestra.run.smithi053.stdout:(71/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 2.3 MB/s | 49 MB 00:21 2024-04-28T12:36:10.747 INFO:teuthology.orchestra.run.smithi191.stdout:(73/152): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-04-28T12:36:10.761 INFO:teuthology.orchestra.run.smithi077.stdout:(42/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 3.9 MB/s | 49 MB 00:12 2024-04-28T12:36:10.831 INFO:teuthology.orchestra.run.smithi191.stdout:(74/152): ceph-mgr-diskprediction-local-19.0.0- 1.2 MB/s | 7.4 MB 00:06 2024-04-28T12:36:10.901 INFO:teuthology.orchestra.run.smithi191.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-28T12:36:10.903 INFO:teuthology.orchestra.run.smithi077.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 211 kB/s | 253 kB 00:01 2024-04-28T12:36:10.923 INFO:teuthology.orchestra.run.smithi191.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-04-28T12:36:10.966 INFO:teuthology.orchestra.run.smithi191.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 381 kB/s | 26 kB 00:00 2024-04-28T12:36:10.982 INFO:teuthology.orchestra.run.smithi191.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 326 kB/s | 19 kB 00:00 2024-04-28T12:36:11.024 INFO:teuthology.orchestra.run.smithi191.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 834 kB/s | 35 kB 00:00 2024-04-28T12:36:11.050 INFO:teuthology.orchestra.run.smithi191.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-28T12:36:11.057 INFO:teuthology.orchestra.run.smithi053.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 206 kB/s | 87 kB 00:00 2024-04-28T12:36:11.133 INFO:teuthology.orchestra.run.smithi191.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-28T12:36:11.207 INFO:teuthology.orchestra.run.smithi053.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 656 kB/s | 1.0 MB 00:01 2024-04-28T12:36:11.217 INFO:teuthology.orchestra.run.smithi191.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-04-28T12:36:11.284 INFO:teuthology.orchestra.run.smithi191.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-28T12:36:11.351 INFO:teuthology.orchestra.run.smithi191.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-28T12:36:11.393 INFO:teuthology.orchestra.run.smithi191.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 390 kB/s | 42 kB 00:00 2024-04-28T12:36:11.435 INFO:teuthology.orchestra.run.smithi191.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 269 kB 00:00 2024-04-28T12:36:11.485 INFO:teuthology.orchestra.run.smithi191.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-28T12:36:11.535 INFO:teuthology.orchestra.run.smithi191.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-04-28T12:36:11.536 INFO:teuthology.orchestra.run.smithi077.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 142 kB/s | 106 kB 00:00 2024-04-28T12:36:11.558 INFO:teuthology.orchestra.run.smithi053.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 697 kB/s | 244 kB 00:00 2024-04-28T12:36:11.561 INFO:teuthology.orchestra.run.smithi191.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-04-28T12:36:11.611 INFO:teuthology.orchestra.run.smithi191.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 713 kB/s | 54 kB 00:00 2024-04-28T12:36:11.670 INFO:teuthology.orchestra.run.smithi191.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-04-28T12:36:11.695 INFO:teuthology.orchestra.run.smithi077.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 171 kB/s | 135 kB 00:00 2024-04-28T12:36:11.728 INFO:teuthology.orchestra.run.smithi191.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 759 kB/s | 44 kB 00:00 2024-04-28T12:36:11.803 INFO:teuthology.orchestra.run.smithi191.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-28T12:36:11.854 INFO:teuthology.orchestra.run.smithi191.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-28T12:36:11.910 INFO:teuthology.orchestra.run.smithi053.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 709 kB/s | 248 kB 00:00 2024-04-28T12:36:11.921 INFO:teuthology.orchestra.run.smithi191.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 710 kB/s | 47 kB 00:00 2024-04-28T12:36:12.005 INFO:teuthology.orchestra.run.smithi191.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-28T12:36:12.016 INFO:teuthology.orchestra.run.smithi053.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 237 kB/s | 26 kB 00:00 2024-04-28T12:36:12.072 INFO:teuthology.orchestra.run.smithi191.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 865 kB/s | 58 kB 00:00 2024-04-28T12:36:12.125 INFO:teuthology.orchestra.run.smithi053.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 177 kB/s | 19 kB 00:00 2024-04-28T12:36:12.255 INFO:teuthology.orchestra.run.smithi191.stdout:(98/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-04-28T12:36:12.304 INFO:teuthology.orchestra.run.smithi077.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 163 kB/s | 125 kB 00:00 2024-04-28T12:36:12.305 INFO:teuthology.orchestra.run.smithi191.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 401 kB/s | 94 kB 00:00 2024-04-28T12:36:12.331 INFO:teuthology.orchestra.run.smithi191.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 849 kB/s | 64 kB 00:00 2024-04-28T12:36:12.390 INFO:teuthology.orchestra.run.smithi191.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 533 kB/s | 45 kB 00:00 2024-04-28T12:36:12.442 INFO:teuthology.orchestra.run.smithi053.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 544 kB/s | 172 kB 00:00 2024-04-28T12:36:12.479 INFO:teuthology.orchestra.run.smithi077.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 689 kB 00:00 2024-04-28T12:36:12.550 INFO:teuthology.orchestra.run.smithi053.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 322 kB/s | 35 kB 00:00 2024-04-28T12:36:12.554 INFO:teuthology.orchestra.run.smithi077.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-28T12:36:12.596 INFO:teuthology.orchestra.run.smithi077.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-04-28T12:36:12.697 INFO:teuthology.orchestra.run.smithi077.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-04-28T12:36:12.730 INFO:teuthology.orchestra.run.smithi077.stdout:(51/152): python3-urllib3-1.26.5-5.el9.noarch.r 207 kB/s | 215 kB 00:01 2024-04-28T12:36:12.756 INFO:teuthology.orchestra.run.smithi077.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-28T12:36:12.789 INFO:teuthology.orchestra.run.smithi077.stdout:(53/152): python3-cryptography-36.0.1-4.el9.x86 472 kB/s | 1.2 MB 00:02 2024-04-28T12:36:12.823 INFO:teuthology.orchestra.run.smithi077.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 355 kB/s | 33 kB 00:00 2024-04-28T12:36:12.865 INFO:teuthology.orchestra.run.smithi077.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 266 kB/s | 11 kB 00:00 2024-04-28T12:36:12.890 INFO:teuthology.orchestra.run.smithi077.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 181 kB/s | 18 kB 00:00 2024-04-28T12:36:12.940 INFO:teuthology.orchestra.run.smithi191.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-04-28T12:36:12.966 INFO:teuthology.orchestra.run.smithi077.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-28T12:36:12.991 INFO:teuthology.orchestra.run.smithi077.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-04-28T12:36:13.016 INFO:teuthology.orchestra.run.smithi077.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2024-04-28T12:36:13.050 INFO:teuthology.orchestra.run.smithi077.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-04-28T12:36:13.075 INFO:teuthology.orchestra.run.smithi077.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-04-28T12:36:13.100 INFO:teuthology.orchestra.run.smithi077.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-28T12:36:13.107 INFO:teuthology.orchestra.run.smithi191.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 3.2 MB/s | 548 kB 00:00 2024-04-28T12:36:13.134 INFO:teuthology.orchestra.run.smithi077.stdout:(63/152): libtool-2.4.6-45.el9.x86_64.rpm 4.0 MB/s | 582 kB 00:00 2024-04-28T12:36:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-04-28T12:36:13.185 INFO:teuthology.orchestra.run.smithi077.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-28T12:36:13.199 INFO:teuthology.orchestra.run.smithi191.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-04-28T12:36:13.218 INFO:teuthology.orchestra.run.smithi077.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-04-28T12:36:13.241 INFO:teuthology.orchestra.run.smithi191.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-04-28T12:36:13.309 INFO:teuthology.orchestra.run.smithi053.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 2.6 MB/s | 6.0 MB 00:02 2024-04-28T12:36:13.335 INFO:teuthology.orchestra.run.smithi077.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2024-04-28T12:36:13.402 INFO:teuthology.orchestra.run.smithi077.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-04-28T12:36:13.418 INFO:teuthology.orchestra.run.smithi053.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-28T12:36:13.444 INFO:teuthology.orchestra.run.smithi077.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-04-28T12:36:13.469 INFO:teuthology.orchestra.run.smithi077.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-28T12:36:13.503 INFO:teuthology.orchestra.run.smithi077.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 5.3 MB/s | 911 kB 00:00 2024-04-28T12:36:13.526 INFO:teuthology.orchestra.run.smithi053.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-28T12:36:13.545 INFO:teuthology.orchestra.run.smithi077.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-28T12:36:13.570 INFO:teuthology.orchestra.run.smithi077.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-28T12:36:13.627 INFO:teuthology.orchestra.run.smithi053.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 771 kB/s | 77 kB 00:00 2024-04-28T12:36:13.629 INFO:teuthology.orchestra.run.smithi077.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-28T12:36:13.658 INFO:teuthology.orchestra.run.smithi191.stdout:(106/152): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-28T12:36:13.699 INFO:teuthology.orchestra.run.smithi191.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-04-28T12:36:13.704 INFO:teuthology.orchestra.run.smithi077.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-28T12:36:13.729 INFO:teuthology.orchestra.run.smithi077.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-28T12:36:13.735 INFO:teuthology.orchestra.run.smithi053.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 391 kB/s | 42 kB 00:00 2024-04-28T12:36:13.741 INFO:teuthology.orchestra.run.smithi191.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-28T12:36:13.746 INFO:teuthology.orchestra.run.smithi077.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 615 kB/s | 26 kB 00:00 2024-04-28T12:36:13.772 INFO:teuthology.orchestra.run.smithi077.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 456 kB/s | 19 kB 00:00 2024-04-28T12:36:13.783 INFO:teuthology.orchestra.run.smithi191.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-28T12:36:13.797 INFO:teuthology.orchestra.run.smithi077.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-28T12:36:13.814 INFO:teuthology.orchestra.run.smithi077.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 832 kB/s | 35 kB 00:00 2024-04-28T12:36:13.825 INFO:teuthology.orchestra.run.smithi191.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-04-28T12:36:13.844 INFO:teuthology.orchestra.run.smithi053.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-04-28T12:36:13.898 INFO:teuthology.orchestra.run.smithi077.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-28T12:36:13.909 INFO:teuthology.orchestra.run.smithi191.stdout:(111/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 839 kB 00:00 2024-04-28T12:36:13.923 INFO:teuthology.orchestra.run.smithi077.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-28T12:36:13.949 INFO:teuthology.orchestra.run.smithi077.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-28T12:36:13.969 INFO:teuthology.orchestra.run.smithi053.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-28T12:36:13.974 INFO:teuthology.orchestra.run.smithi077.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-28T12:36:13.976 INFO:teuthology.orchestra.run.smithi191.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-28T12:36:13.999 INFO:teuthology.orchestra.run.smithi077.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 42 kB 00:00 2024-04-28T12:36:14.024 INFO:teuthology.orchestra.run.smithi077.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-28T12:36:14.034 INFO:teuthology.orchestra.run.smithi191.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 748 kB/s | 43 kB 00:00 2024-04-28T12:36:14.050 INFO:teuthology.orchestra.run.smithi077.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-28T12:36:14.078 INFO:teuthology.orchestra.run.smithi053.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-28T12:36:14.083 INFO:teuthology.orchestra.run.smithi077.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-04-28T12:36:14.085 INFO:teuthology.orchestra.run.smithi191.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 645 kB/s | 32 kB 00:00 2024-04-28T12:36:14.109 INFO:teuthology.orchestra.run.smithi077.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 872 kB/s | 51 kB 00:00 2024-04-28T12:36:14.127 INFO:teuthology.orchestra.run.smithi191.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-04-28T12:36:14.134 INFO:teuthology.orchestra.run.smithi077.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-28T12:36:14.169 INFO:teuthology.orchestra.run.smithi191.stdout:(116/152): protobuf-compiler-3.14.0-13.el9.x86_ 485 kB/s | 863 kB 00:01 2024-04-28T12:36:14.187 INFO:teuthology.orchestra.run.smithi053.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 470 kB/s | 51 kB 00:00 2024-04-28T12:36:14.194 INFO:teuthology.orchestra.run.smithi191.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-28T12:36:14.209 INFO:teuthology.orchestra.run.smithi077.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-28T12:36:14.235 INFO:teuthology.orchestra.run.smithi077.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-04-28T12:36:14.252 INFO:teuthology.orchestra.run.smithi191.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-28T12:36:14.260 INFO:teuthology.orchestra.run.smithi077.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-04-28T12:36:14.285 INFO:teuthology.orchestra.run.smithi077.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-28T12:36:14.295 INFO:teuthology.orchestra.run.smithi191.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 846 kB/s | 35 kB 00:00 2024-04-28T12:36:14.295 INFO:teuthology.orchestra.run.smithi053.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 495 kB/s | 54 kB 00:00 2024-04-28T12:36:14.310 INFO:teuthology.orchestra.run.smithi077.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-28T12:36:14.336 INFO:teuthology.orchestra.run.smithi077.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-04-28T12:36:14.345 INFO:teuthology.orchestra.run.smithi191.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-28T12:36:14.369 INFO:teuthology.orchestra.run.smithi077.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-28T12:36:14.378 INFO:teuthology.orchestra.run.smithi191.stdout:(121/152): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-04-28T12:36:14.394 INFO:teuthology.orchestra.run.smithi077.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 990 kB/s | 58 kB 00:00 2024-04-28T12:36:14.420 INFO:teuthology.orchestra.run.smithi077.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-28T12:36:14.445 INFO:teuthology.orchestra.run.smithi077.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-28T12:36:14.446 INFO:teuthology.orchestra.run.smithi191.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-28T12:36:14.470 INFO:teuthology.orchestra.run.smithi077.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 896 kB/s | 45 kB 00:00 2024-04-28T12:36:14.487 INFO:teuthology.orchestra.run.smithi191.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-04-28T12:36:14.537 INFO:teuthology.orchestra.run.smithi191.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-28T12:36:14.563 INFO:teuthology.orchestra.run.smithi191.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-04-28T12:36:14.590 INFO:teuthology.orchestra.run.smithi191.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-04-28T12:36:14.613 INFO:teuthology.orchestra.run.smithi191.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-04-28T12:36:14.639 INFO:teuthology.orchestra.run.smithi191.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-28T12:36:14.681 INFO:teuthology.orchestra.run.smithi191.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-28T12:36:14.723 INFO:teuthology.orchestra.run.smithi191.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-28T12:36:14.765 INFO:teuthology.orchestra.run.smithi191.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-28T12:36:14.815 INFO:teuthology.orchestra.run.smithi191.stdout:(132/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-28T12:36:14.846 INFO:teuthology.orchestra.run.smithi077.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-28T12:36:14.865 INFO:teuthology.orchestra.run.smithi191.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 4.1 MB/s | 1.0 MB 00:00 2024-04-28T12:36:14.882 INFO:teuthology.orchestra.run.smithi191.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-04-28T12:36:14.907 INFO:teuthology.orchestra.run.smithi191.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-04-28T12:36:14.924 INFO:teuthology.orchestra.run.smithi191.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 673 kB/s | 28 kB 00:00 2024-04-28T12:36:14.938 INFO:teuthology.orchestra.run.smithi077.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 45 kB/s | 22 kB 00:00 2024-04-28T12:36:14.941 INFO:teuthology.orchestra.run.smithi191.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 918 kB/s | 31 kB 00:00 2024-04-28T12:36:14.975 INFO:teuthology.orchestra.run.smithi191.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-28T12:36:14.992 INFO:teuthology.orchestra.run.smithi191.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-28T12:36:15.009 INFO:teuthology.orchestra.run.smithi191.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-04-28T12:36:15.026 INFO:teuthology.orchestra.run.smithi191.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-04-28T12:36:15.051 INFO:teuthology.orchestra.run.smithi191.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-04-28T12:36:15.076 INFO:teuthology.orchestra.run.smithi191.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-28T12:36:15.102 INFO:teuthology.orchestra.run.smithi191.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-04-28T12:36:15.127 INFO:teuthology.orchestra.run.smithi191.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-04-28T12:36:15.153 INFO:teuthology.orchestra.run.smithi191.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-28T12:36:15.222 INFO:teuthology.orchestra.run.smithi077.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-28T12:36:15.263 INFO:teuthology.orchestra.run.smithi077.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 469 kB/s | 19 kB 00:00 2024-04-28T12:36:15.339 INFO:teuthology.orchestra.run.smithi077.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-28T12:36:15.380 INFO:teuthology.orchestra.run.smithi077.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-04-28T12:36:15.402 INFO:teuthology.orchestra.run.smithi191.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 5.7 MB/s | 1.6 MB 00:00 2024-04-28T12:36:15.422 INFO:teuthology.orchestra.run.smithi077.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-28T12:36:15.490 INFO:teuthology.orchestra.run.smithi077.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-28T12:36:15.615 INFO:teuthology.orchestra.run.smithi077.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-28T12:36:15.717 INFO:teuthology.orchestra.run.smithi077.stdout:(111/152): protobuf-compiler-3.14.0-13.el9.x86_ 693 kB/s | 863 kB 00:01 2024-04-28T12:36:16.009 INFO:teuthology.orchestra.run.smithi077.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.1 MB/s | 839 kB 00:00 2024-04-28T12:36:16.043 INFO:teuthology.orchestra.run.smithi077.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-28T12:36:16.077 INFO:teuthology.orchestra.run.smithi077.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 972 kB/s | 32 kB 00:00 2024-04-28T12:36:16.110 INFO:teuthology.orchestra.run.smithi077.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-04-28T12:36:16.135 INFO:teuthology.orchestra.run.smithi077.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-04-28T12:36:16.202 INFO:teuthology.orchestra.run.smithi077.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-28T12:36:16.311 INFO:teuthology.orchestra.run.smithi077.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.0 MB/s | 358 kB 00:00 2024-04-28T12:36:16.336 INFO:teuthology.orchestra.run.smithi077.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-28T12:36:16.362 INFO:teuthology.orchestra.run.smithi077.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-04-28T12:36:16.437 INFO:teuthology.orchestra.run.smithi077.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-28T12:36:16.520 INFO:teuthology.orchestra.run.smithi077.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-28T12:36:16.562 INFO:teuthology.orchestra.run.smithi077.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-04-28T12:36:16.605 INFO:teuthology.orchestra.run.smithi077.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 424 kB/s | 18 kB 00:00 2024-04-28T12:36:16.646 INFO:teuthology.orchestra.run.smithi077.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-04-28T12:36:16.662 INFO:teuthology.orchestra.run.smithi053.stdout:(90/152): postgresql-server-13.14-1.el9.x86_64. 742 kB/s | 5.8 MB 00:07 2024-04-28T12:36:16.688 INFO:teuthology.orchestra.run.smithi077.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 467 kB/s | 19 kB 00:00 2024-04-28T12:36:16.730 INFO:teuthology.orchestra.run.smithi077.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 474 kB/s | 20 kB 00:00 2024-04-28T12:36:16.762 INFO:teuthology.orchestra.run.smithi053.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 418 kB/s | 42 kB 00:00 2024-04-28T12:36:16.862 INFO:teuthology.orchestra.run.smithi053.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-04-28T12:36:17.006 INFO:teuthology.orchestra.run.smithi077.stdout:(128/152): libarrow-9.0.0-11.el9.x86_64.rpm 2.5 MB/s | 4.4 MB 00:01 2024-04-28T12:36:17.050 INFO:teuthology.orchestra.run.smithi077.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-28T12:36:17.098 INFO:teuthology.orchestra.run.smithi077.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 2.8 MB/s | 1.0 MB 00:00 2024-04-28T12:36:17.140 INFO:teuthology.orchestra.run.smithi077.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 860 kB/s | 79 kB 00:00 2024-04-28T12:36:17.165 INFO:teuthology.orchestra.run.smithi077.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 865 kB/s | 58 kB 00:00 2024-04-28T12:36:17.186 INFO:teuthology.orchestra.run.smithi191.stdout:(148/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-04-28T12:36:17.204 INFO:teuthology.orchestra.run.smithi053.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 646 kB/s | 221 kB 00:00 2024-04-28T12:36:17.219 INFO:teuthology.orchestra.run.smithi077.stdout:(133/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.4 MB/s | 2.0 MB 00:00 2024-04-28T12:36:17.249 INFO:teuthology.orchestra.run.smithi077.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-04-28T12:36:17.291 INFO:teuthology.orchestra.run.smithi077.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-28T12:36:17.305 INFO:teuthology.orchestra.run.smithi053.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 359 kB/s | 36 kB 00:00 2024-04-28T12:36:17.317 INFO:teuthology.orchestra.run.smithi077.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-04-28T12:36:17.337 INFO:teuthology.orchestra.run.smithi191.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 1.3 MB 00:00 2024-04-28T12:36:17.342 INFO:teuthology.orchestra.run.smithi077.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-04-28T12:36:17.367 INFO:teuthology.orchestra.run.smithi077.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-28T12:36:17.393 INFO:teuthology.orchestra.run.smithi077.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-28T12:36:17.396 INFO:teuthology.orchestra.run.smithi053.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 517 kB/s | 47 kB 00:00 2024-04-28T12:36:17.418 INFO:teuthology.orchestra.run.smithi077.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-04-28T12:36:17.420 INFO:teuthology.orchestra.run.smithi191.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-28T12:36:17.443 INFO:teuthology.orchestra.run.smithi077.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-28T12:36:17.510 INFO:teuthology.orchestra.run.smithi077.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-28T12:36:17.544 INFO:teuthology.orchestra.run.smithi077.stdout:(143/152): python3-routes-2.5.1-5.el9.noarch.rp 937 kB/s | 188 kB 00:00 2024-04-28T12:36:17.577 INFO:teuthology.orchestra.run.smithi077.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 601 kB/s | 20 kB 00:00 2024-04-28T12:36:17.644 INFO:teuthology.orchestra.run.smithi077.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-04-28T12:36:17.736 INFO:teuthology.orchestra.run.smithi077.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-28T12:36:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 610 kB/s | 305 kB 00:00 2024-04-28T12:36:17.987 INFO:teuthology.orchestra.run.smithi077.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 897 kB/s | 427 kB 00:00 2024-04-28T12:36:18.047 INFO:teuthology.orchestra.run.smithi053.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 385 kB/s | 58 kB 00:00 2024-04-28T12:36:18.264 INFO:teuthology.orchestra.run.smithi053.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 432 kB/s | 94 kB 00:00 2024-04-28T12:36:18.465 INFO:teuthology.orchestra.run.smithi053.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 318 kB/s | 64 kB 00:00 2024-04-28T12:36:18.565 INFO:teuthology.orchestra.run.smithi053.stdout:(100/152): python3-numpy-1.20.1-5.el9.x86_64.rp 868 kB/s | 5.1 MB 00:06 2024-04-28T12:36:18.581 INFO:teuthology.orchestra.run.smithi053.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 384 kB/s | 45 kB 00:00 2024-04-28T12:36:18.690 INFO:teuthology.orchestra.run.smithi053.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-04-28T12:36:18.745 INFO:teuthology.orchestra.run.smithi077.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-04-28T12:36:18.816 INFO:teuthology.orchestra.run.smithi053.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-04-28T12:36:19.154 INFO:teuthology.orchestra.run.smithi191.stdout:(151/152): librados2-19.0.0-3306.gbc612ffb.el9. 869 kB/s | 3.4 MB 00:04 2024-04-28T12:36:19.262 INFO:teuthology.orchestra.run.smithi077.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 2.5 MB/s | 3.1 MB 00:01 2024-04-28T12:36:19.274 INFO:teuthology.orchestra.run.smithi053.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 939 kB/s | 548 kB 00:00 2024-04-28T12:36:19.313 INFO:teuthology.orchestra.run.smithi077.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 2.2 MB/s | 3.4 MB 00:01 2024-04-28T12:36:19.424 INFO:teuthology.orchestra.run.smithi053.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 507 kB/s | 309 kB 00:00 2024-04-28T12:36:19.450 INFO:teuthology.orchestra.run.smithi053.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-04-28T12:36:19.550 INFO:teuthology.orchestra.run.smithi053.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-28T12:36:19.650 INFO:teuthology.orchestra.run.smithi053.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-04-28T12:36:19.767 INFO:teuthology.orchestra.run.smithi053.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2024-04-28T12:36:19.910 INFO:teuthology.orchestra.run.smithi053.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-04-28T12:36:19.935 INFO:teuthology.orchestra.run.smithi053.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 904 kB/s | 151 kB 00:00 2024-04-28T12:36:20.171 INFO:teuthology.orchestra.run.smithi053.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.1 MB/s | 839 kB 00:00 2024-04-28T12:36:20.210 INFO:teuthology.orchestra.run.smithi053.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-04-28T12:36:20.269 INFO:teuthology.orchestra.run.smithi053.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 444 kB/s | 43 kB 00:00 2024-04-28T12:36:20.302 INFO:teuthology.orchestra.run.smithi053.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-04-28T12:36:20.361 INFO:teuthology.orchestra.run.smithi053.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-28T12:36:20.411 INFO:teuthology.orchestra.run.smithi053.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-28T12:36:20.462 INFO:teuthology.orchestra.run.smithi053.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-28T12:36:20.537 INFO:teuthology.orchestra.run.smithi053.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-28T12:36:20.562 INFO:teuthology.orchestra.run.smithi053.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-28T12:36:20.721 INFO:teuthology.orchestra.run.smithi053.stdout:(121/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.5 MB/s | 16 MB 00:06 2024-04-28T12:36:20.746 INFO:teuthology.orchestra.run.smithi053.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 986 kB/s | 206 kB 00:00 2024-04-28T12:36:20.805 INFO:teuthology.orchestra.run.smithi053.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-04-28T12:36:20.838 INFO:teuthology.orchestra.run.smithi053.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-28T12:36:20.897 INFO:teuthology.orchestra.run.smithi053.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-28T12:36:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-28T12:36:20.989 INFO:teuthology.orchestra.run.smithi053.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-28T12:36:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-28T12:36:21.106 INFO:teuthology.orchestra.run.smithi053.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-28T12:36:21.121 INFO:teuthology.orchestra.run.smithi077.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 582 kB/s | 1.3 MB 00:02 2024-04-28T12:36:21.134 INFO:teuthology.orchestra.run.smithi053.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2024-04-28T12:36:21.173 INFO:teuthology.orchestra.run.smithi053.stdout:(131/152): python3-grpcio-tools-1.46.7-10.el9.x 319 kB/s | 144 kB 00:00 2024-04-28T12:36:21.198 INFO:teuthology.orchestra.run.smithi053.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-04-28T12:36:21.232 INFO:teuthology.orchestra.run.smithi053.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 573 kB/s | 58 kB 00:00 2024-04-28T12:36:21.304 INFO:teuthology.orchestra.run.smithi191.stdout:(152/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 2.1 MB/s | 49 MB 00:23 2024-04-28T12:36:21.307 INFO:teuthology.orchestra.run.smithi053.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-28T12:36:21.308 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:36:21.308 INFO:teuthology.orchestra.run.smithi191.stdout:Total 5.8 MB/s | 222 MB 00:37 2024-04-28T12:36:21.333 INFO:teuthology.orchestra.run.smithi053.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-04-28T12:36:21.375 INFO:teuthology.orchestra.run.smithi053.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-28T12:36:21.392 INFO:teuthology.orchestra.run.smithi053.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-04-28T12:36:21.417 INFO:teuthology.orchestra.run.smithi053.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-04-28T12:36:21.484 INFO:teuthology.orchestra.run.smithi053.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-04-28T12:36:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-04-28T12:36:21.526 INFO:teuthology.orchestra.run.smithi053.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 328 kB/s | 36 kB 00:00 2024-04-28T12:36:21.568 INFO:teuthology.orchestra.run.smithi053.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-04-28T12:36:21.627 INFO:teuthology.orchestra.run.smithi053.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-04-28T12:36:21.653 INFO:teuthology.orchestra.run.smithi053.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-04-28T12:36:21.677 INFO:teuthology.orchestra.run.smithi053.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-28T12:36:21.719 INFO:teuthology.orchestra.run.smithi053.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-04-28T12:36:21.803 INFO:teuthology.orchestra.run.smithi053.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-28T12:36:21.828 INFO:teuthology.orchestra.run.smithi053.stdout:(148/152): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-28T12:36:21.937 INFO:teuthology.orchestra.run.smithi053.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 12 MB/s | 1.3 MB 00:00 2024-04-28T12:36:22.154 INFO:teuthology.orchestra.run.smithi053.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-04-28T12:36:22.412 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-04-28T12:36:22.554 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-04-28T12:36:22.555 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-04-28T12:36:22.854 INFO:teuthology.orchestra.run.smithi053.stdout:(151/152): librados2-19.0.0-3306.gbc612ffb.el9. 3.0 MB/s | 3.4 MB 00:01 2024-04-28T12:36:22.947 INFO:teuthology.orchestra.run.smithi077.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 665 kB/s | 2.4 MB 00:03 2024-04-28T12:36:22.954 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:36:22.954 INFO:teuthology.orchestra.run.smithi077.stdout:Total 5.8 MB/s | 222 MB 00:37 2024-04-28T12:36:23.337 INFO:teuthology.orchestra.run.smithi053.stdout:(152/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 2.0 MB/s | 3.1 MB 00:01 2024-04-28T12:36:23.341 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:36:23.341 INFO:teuthology.orchestra.run.smithi053.stdout:Total 5.1 MB/s | 222 MB 00:43 2024-04-28T12:36:24.069 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-04-28T12:36:24.211 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-04-28T12:36:24.211 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-04-28T12:36:24.447 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-04-28T12:36:24.447 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-04-28T12:36:24.492 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-04-28T12:36:24.652 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-04-28T12:36:24.652 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-04-28T12:36:25.971 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-04-28T12:36:25.972 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-04-28T12:36:26.540 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-04-28T12:36:26.540 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-04-28T12:36:30.407 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-04-28T12:36:30.490 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T12:36:30.529 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T12:36:30.582 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T12:36:30.616 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T12:36:30.642 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T12:36:30.684 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T12:36:31.012 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T12:36:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:36:31.240 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:36:31.269 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:36:31.360 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:36:31.390 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:36:31.544 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T12:36:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T12:36:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:36:31.710 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:36:31.733 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:36:31.795 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:36:31.817 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:36:31.838 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:36:31.960 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-04-28T12:36:31.992 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:36:32.034 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:36:32.059 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T12:36:32.097 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T12:36:32.141 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T12:36:32.151 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T12:36:32.185 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T12:36:32.210 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T12:36:32.230 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T12:36:32.253 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T12:36:32.266 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T12:36:32.301 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T12:36:32.366 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T12:36:32.400 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T12:36:32.444 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T12:36:32.477 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T12:36:32.504 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T12:36:32.573 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-04-28T12:36:32.574 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T12:36:32.588 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T12:36:32.658 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T12:36:32.697 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T12:36:32.703 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:36:32.751 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T12:36:32.792 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T12:36:32.815 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:36:32.817 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T12:36:32.837 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T12:36:32.846 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:36:32.860 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T12:36:32.874 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T12:36:32.918 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T12:36:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:36:32.950 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:36:32.995 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T12:36:33.029 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T12:36:33.072 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T12:36:33.101 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T12:36:33.106 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T12:36:33.164 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T12:36:33.172 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T12:36:33.192 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:36:33.206 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T12:36:33.215 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T12:36:33.294 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:36:33.300 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T12:36:33.318 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:36:33.326 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:36:33.380 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:36:33.402 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:36:33.426 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:36:33.434 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:36:33.442 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T12:36:33.461 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:36:33.494 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T12:36:33.537 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:36:33.565 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:36:33.602 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:36:33.635 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:36:33.721 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T12:36:33.742 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T12:36:33.778 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T12:36:33.807 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:36:33.837 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T12:36:33.868 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T12:36:33.902 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T12:36:33.912 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:36:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:36:33.967 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T12:36:34.003 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:36:34.010 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T12:36:34.025 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:36:34.030 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T12:36:34.052 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:36:34.053 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T12:36:34.086 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T12:36:34.114 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T12:36:34.115 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T12:36:34.143 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T12:36:34.182 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T12:36:34.186 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T12:36:34.225 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T12:36:34.236 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T12:36:34.251 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:36:34.266 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T12:36:34.267 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T12:36:34.300 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:36:34.302 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T12:36:34.386 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T12:36:34.412 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T12:36:34.414 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T12:36:34.430 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T12:36:34.473 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T12:36:34.494 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T12:36:34.502 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T12:36:34.506 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T12:36:34.549 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T12:36:34.550 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T12:36:34.579 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T12:36:34.584 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T12:36:34.622 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T12:36:34.625 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T12:36:34.653 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T12:36:34.659 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T12:36:34.690 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T12:36:34.699 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T12:36:34.700 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T12:36:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T12:36:34.744 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T12:36:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T12:36:34.776 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T12:36:34.804 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T12:36:34.816 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T12:36:34.832 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T12:36:34.858 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T12:36:34.867 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T12:36:34.903 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T12:36:34.919 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T12:36:34.948 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T12:36:34.984 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T12:36:34.996 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T12:36:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T12:36:35.061 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T12:36:35.069 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T12:36:35.094 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T12:36:35.111 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T12:36:35.138 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T12:36:35.166 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T12:36:35.206 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T12:36:35.241 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T12:36:35.244 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T12:36:35.284 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T12:36:35.289 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T12:36:35.310 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T12:36:35.334 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T12:36:35.367 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T12:36:35.410 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T12:36:35.417 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T12:36:35.498 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T12:36:35.511 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T12:36:35.528 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T12:36:35.537 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T12:36:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T12:36:35.571 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T12:36:35.576 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T12:36:35.595 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T12:36:35.609 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T12:36:35.630 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T12:36:35.651 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T12:36:35.669 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T12:36:35.700 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T12:36:35.730 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T12:36:35.765 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T12:36:35.798 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T12:36:35.820 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T12:36:35.848 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T12:36:35.879 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T12:36:35.882 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T12:36:35.933 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T12:36:35.970 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T12:36:36.006 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T12:36:36.013 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T12:36:36.038 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T12:36:36.045 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T12:36:36.074 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T12:36:36.092 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T12:36:36.121 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T12:36:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T12:36:36.157 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T12:36:36.201 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T12:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T12:36:36.216 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T12:36:36.241 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T12:36:36.242 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T12:36:36.273 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T12:36:36.284 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T12:36:36.322 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T12:36:36.364 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T12:36:36.378 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T12:36:36.424 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T12:36:36.462 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T12:36:36.504 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T12:36:36.522 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T12:36:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T12:36:36.597 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T12:36:36.637 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T12:36:36.640 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T12:36:36.677 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T12:36:36.711 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T12:36:36.714 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T12:36:36.736 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T12:36:36.750 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T12:36:36.787 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T12:36:36.810 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T12:36:36.835 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T12:36:36.869 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T12:36:36.913 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T12:36:36.938 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T12:36:36.955 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T12:36:36.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T12:36:36.997 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T12:36:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T12:36:37.039 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T12:36:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T12:36:37.087 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T12:36:37.100 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T12:36:37.102 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T12:36:37.127 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T12:36:37.143 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T12:36:37.157 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T12:36:37.173 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T12:36:37.191 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T12:36:37.203 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T12:36:37.215 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T12:36:37.225 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T12:36:37.253 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T12:36:37.267 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T12:36:37.301 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T12:36:37.313 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T12:36:37.341 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T12:36:37.356 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T12:36:37.380 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T12:36:37.388 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T12:36:37.388 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T12:36:37.413 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T12:36:37.431 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T12:36:37.438 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T12:36:37.463 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T12:36:37.517 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T12:36:37.518 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T12:36:37.557 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T12:36:37.586 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T12:36:37.624 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T12:36:37.646 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T12:36:37.664 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T12:36:37.678 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T12:36:37.681 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T12:36:37.707 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T12:36:37.742 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T12:36:37.781 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T12:36:37.821 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T12:36:37.850 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T12:36:37.885 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T12:36:37.918 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T12:36:37.961 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T12:36:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T12:36:38.326 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T12:36:38.336 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T12:36:38.383 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T12:36:38.405 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T12:36:38.414 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T12:36:38.436 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T12:36:38.453 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T12:36:38.462 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T12:36:38.487 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T12:36:38.572 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T12:36:38.593 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T12:36:38.633 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T12:36:38.653 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T12:36:38.758 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T12:36:38.790 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T12:36:38.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T12:36:39.004 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T12:36:39.083 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T12:36:39.397 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T12:36:39.437 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T12:36:39.471 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T12:36:39.575 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T12:36:39.630 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T12:36:39.679 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T12:36:39.720 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T12:36:39.722 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T12:36:39.758 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T12:36:39.764 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T12:36:39.786 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T12:36:39.828 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T12:36:39.835 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T12:36:39.951 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T12:36:39.978 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T12:36:40.059 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T12:36:40.138 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T12:36:40.182 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:36:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:36:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T12:36:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T12:36:40.747 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T12:36:40.818 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T12:36:40.852 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:36:40.855 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T12:36:40.923 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:36:40.953 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T12:36:41.051 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T12:36:41.143 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T12:36:41.170 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T12:36:41.204 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T12:36:41.535 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T12:36:41.592 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:36:41.682 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:36:41.714 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T12:36:41.750 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T12:36:42.164 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T12:36:42.231 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T12:36:42.249 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T12:36:42.261 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:36:42.277 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T12:36:42.310 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T12:36:42.325 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:36:42.355 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T12:36:42.639 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T12:36:42.681 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:36:42.764 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:36:42.794 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T12:36:42.830 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T12:36:43.256 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:43.293 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:43.337 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T12:36:43.358 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:36:43.413 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:36:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T12:36:43.837 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:43.881 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:36:43.965 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:36:43.966 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T12:36:43.966 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T12:36:43.966 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:36:43.998 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:36:44.626 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:44.660 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:45.203 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:45.247 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:36:45.328 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:36:45.328 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T12:36:45.328 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T12:36:45.328 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:36:45.357 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:36:45.802 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:45.832 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:46.403 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:36:46.443 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:36:46.539 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:36:46.539 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T12:36:46.539 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T12:36:46.539 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:36:46.578 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-04-28T12:36:55.812 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-04-28T12:36:55.813 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:36:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:36:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:36:56.003 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:56.003 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T12:36:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:36:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:36:56.651 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-04-28T12:36:56.652 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-04-28T12:36:56.652 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-04-28T12:36:56.652 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-04-28T12:36:56.652 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-04-28T12:36:56.652 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-04-28T12:36:56.652 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-04-28T12:36:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:36:56.821 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:36:56.837 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:36:56.837 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:56.838 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T12:36:56.838 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:36:57.482 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:36:57.507 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:36:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T12:36:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:36:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:36:57.508 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:36:57.733 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:36:57.755 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:36:57.755 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:57.755 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T12:36:57.755 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:36:57.755 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:36:57.756 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:36:58.200 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:36:58.222 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:36:58.223 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:58.223 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T12:36:58.223 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:36:58.223 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:36:58.223 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:36:58.330 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T12:36:58.332 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:36:58.358 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:36:58.358 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:58.359 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T12:36:58.359 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:36:58.359 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:36:58.359 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:36:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-04-28T12:36:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-04-28T12:36:58.422 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:36:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T12:36:58.583 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:36:58.606 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:36:58.606 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:58.606 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T12:36:58.606 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:36:58.606 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:36:58.606 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:36:58.607 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:36:58.624 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:36:58.624 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:58.624 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T12:36:58.624 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:36:58.639 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T12:36:58.693 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T12:36:58.772 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T12:36:58.868 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:36:59.009 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:36:59.051 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:36:59.051 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T12:36:59.074 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:36:59.074 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:36:59.074 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T12:36:59.074 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:36:59.074 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:36:59.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:36:59.083 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T12:36:59.111 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T12:36:59.181 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T12:36:59.229 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T12:36:59.284 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T12:36:59.478 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T12:36:59.526 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T12:36:59.547 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:36:59.569 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:36:59.595 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T12:36:59.601 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T12:36:59.663 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T12:36:59.684 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:36:59.692 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:36:59.823 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:36:59.869 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T12:36:59.901 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T12:36:59.929 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T12:37:00.161 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:37:00.188 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:37:00.188 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:00.188 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T12:37:00.188 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:37:00.188 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:37:00.188 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:00.372 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:37:00.402 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:37:00.419 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:37:00.434 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T12:37:00.444 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:37:00.444 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:00.444 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T12:37:00.444 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:37:00.444 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:37:00.444 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:00.496 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T12:37:00.543 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:37:00.738 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:37:00.769 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:37:00.903 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:37:00.927 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:37:00.927 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:00.928 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T12:37:00.928 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:37:00.928 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:37:00.928 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T12:37:01.092 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T12:37:01.156 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T12:37:01.352 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T12:37:01.405 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T12:37:01.486 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T12:37:01.575 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:37:01.583 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:37:01.611 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:37:01.726 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:37:01.730 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:37:01.752 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:37:01.763 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T12:37:01.795 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T12:37:01.822 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T12:37:01.855 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:37:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T12:37:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:37:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:37:02.072 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:02.072 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T12:37:02.072 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:37:02.072 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:37:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:37:02.116 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:37:02.128 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:37:02.201 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:37:02.225 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T12:37:02.229 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:37:02.262 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T12:37:02.282 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T12:37:02.334 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T12:37:02.360 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:37:02.571 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:37:02.603 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:37:02.705 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:37:02.864 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T12:37:02.897 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:37:02.920 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:37:02.920 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:02.920 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T12:37:02.920 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:37:02.920 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:37:02.921 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:37:02.965 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:37:02.975 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:37:03.073 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T12:37:03.122 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T12:37:03.202 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T12:37:03.245 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:37:03.267 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:37:03.268 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:03.268 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T12:37:03.268 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:37:03.268 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:37:03.268 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:37:03.329 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:37:03.351 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:37:03.351 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:03.351 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T12:37:03.351 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:37:03.397 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:37:03.419 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:37:03.419 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:03.420 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T12:37:03.420 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:37:03.420 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:37:03.420 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:37:03.459 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:37:03.504 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:37:03.731 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:37:03.754 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:37:03.754 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:03.754 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T12:37:03.754 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:37:03.754 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:37:03.754 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:37:04.043 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T12:37:04.076 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:37:04.099 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:37:04.099 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:04.099 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T12:37:04.099 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:37:04.099 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:37:04.099 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:37:04.143 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:37:04.165 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:37:04.165 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:04.165 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T12:37:04.165 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:37:04.210 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:37:04.233 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:37:04.233 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:04.234 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T12:37:04.234 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:37:04.234 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:37:04.234 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:37:04.520 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:37:04.544 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:37:04.546 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:37:04.567 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:37:04.568 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:04.568 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T12:37:04.568 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:37:04.568 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:37:04.568 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:37:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:37:04.822 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T12:37:04.856 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:37:04.881 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:37:04.881 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:04.881 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T12:37:04.881 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:37:04.881 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:37:04.881 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:04.932 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:37:04.943 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:37:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T12:37:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T12:37:06.082 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T12:37:06.109 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:37:06.133 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:37:06.133 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:06.133 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T12:37:06.133 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:37:06.133 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:37:06.133 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:06.177 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:37:06.200 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:37:06.201 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:06.201 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T12:37:06.201 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:06.252 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:37:06.276 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:37:06.276 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:06.277 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T12:37:06.277 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:37:06.277 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:37:06.277 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:06.603 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:37:06.627 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:37:06.628 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:37:06.628 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T12:37:06.628 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:37:06.628 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:37:06.628 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:08.395 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T12:37:08.436 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T12:37:08.478 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T12:37:08.510 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T12:37:08.538 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T12:37:08.722 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T12:37:08.794 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T12:37:08.824 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T12:37:09.173 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T12:37:09.214 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T12:37:09.256 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T12:37:09.288 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T12:37:09.316 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T12:37:09.452 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:09.483 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:09.496 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T12:37:09.503 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:09.545 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T12:37:09.575 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T12:37:10.140 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:10.182 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:10.201 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:10.336 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T12:37:10.434 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T12:37:10.476 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T12:37:10.559 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T12:37:10.590 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T12:37:10.590 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:37:10.666 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:37:10.699 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:37:10.699 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:37:10.798 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:37:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:11.029 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T12:37:11.121 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T12:37:11.161 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T12:37:11.242 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T12:37:11.283 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T12:37:11.283 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:37:11.359 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:37:11.400 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:37:11.400 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:37:11.490 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T12:37:11.493 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:37:11.532 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T12:37:11.557 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:11.573 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T12:37:11.605 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T12:37:11.632 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T12:37:11.824 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T12:37:11.907 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T12:37:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T12:37:12.519 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:12.555 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:12.575 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:37:13.466 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T12:37:13.552 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T12:37:13.594 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T12:37:13.677 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T12:37:13.706 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T12:37:13.706 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:37:13.790 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:37:13.823 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:37:13.823 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:37:13.915 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:37:13.991 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:17.094 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:17.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T12:37:17.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T12:37:17.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T12:37:17.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T12:37:17.097 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T12:37:17.098 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T12:37:17.099 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T12:37:17.100 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T12:37:17.101 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T12:37:17.102 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T12:37:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T12:37:17.104 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T12:37:17.105 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T12:37:17.106 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T12:37:17.106 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T12:37:17.106 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:37:17.106 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T12:37:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T12:37:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T12:37:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T12:37:17.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T12:37:17.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T12:37:17.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T12:37:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T12:37:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T12:37:17.403 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T12:37:17.404 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T12:37:17.405 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T12:37:17.406 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T12:37:17.407 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T12:37:17.408 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T12:37:17.409 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T12:37:17.410 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T12:37:17.411 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T12:37:17.412 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T12:37:17.413 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T12:37:17.414 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T12:37:17.414 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T12:37:17.414 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:37:17.414 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T12:37:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T12:37:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T12:37:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T12:37:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T12:37:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T12:37:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T12:37:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T12:37:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T12:37:19.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T12:37:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T12:37:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T12:37:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T12:37:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T12:37:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T12:37:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T12:37:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:37:19.200 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T12:37:19.543 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.544 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T12:37:19.545 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.546 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T12:37:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T12:37:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T12:37:19.549 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T12:37:19.550 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T12:37:19.551 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T12:37:19.552 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T12:37:19.553 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T12:37:19.554 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T12:37:19.555 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T12:37:19.555 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-28T12:37:19.555 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-04-28T12:37:19.559 DEBUG:teuthology.parallel:result is None 2024-04-28T12:37:19.851 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:19.851 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T12:37:19.851 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T12:37:19.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T12:37:19.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T12:37:19.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T12:37:19.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T12:37:19.856 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T12:37:19.857 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T12:37:19.858 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T12:37:19.859 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T12:37:19.860 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T12:37:19.861 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T12:37:19.862 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T12:37:19.863 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T12:37:19.863 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:37:19.863 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T12:37:19.936 DEBUG:teuthology.parallel:result is None 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-04-28T12:37:21.939 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.940 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.941 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T12:37:21.942 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.943 INFO:teuthology.orchestra.run.smithi053.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T12:37:21.944 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T12:37:21.945 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T12:37:21.946 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T12:37:21.947 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T12:37:21.948 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T12:37:21.949 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T12:37:21.950 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T12:37:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T12:37:21.952 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T12:37:21.953 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T12:37:21.953 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:37:21.953 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-28T12:37:22.344 DEBUG:teuthology.parallel:result is None 2024-04-28T12:37:22.345 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:37:22.577 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T12:37:22.600 INFO:teuthology.orchestra.run.smithi053.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T12:37:22.601 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T12:37:22.601 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T12:37:22.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:37:22.741 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T12:37:22.767 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T12:37:22.767 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T12:37:22.767 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T12:37:22.769 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:37:22.997 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T12:37:23.023 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T12:37:23.023 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T12:37:23.023 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T12:37:23.025 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-28T12:37:23.025 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:37:23.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T12:37:23.054 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:37:23.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T12:37:23.086 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:37:23.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T12:37:23.116 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-28T12:37:23.116 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:37:23.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T12:37:23.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T12:37:23.207 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:37:23.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T12:37:23.234 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T12:37:23.300 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:37:23.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T12:37:23.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T12:37:23.396 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-28T12:37:23.397 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:37:23.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T12:37:23.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T12:37:23.489 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:37:23.489 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T12:37:23.515 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T12:37:23.579 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:37:23.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T12:37:23.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T12:37:23.676 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-28T12:37:23.676 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:37:23.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T12:37:23.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T12:37:23.766 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:37:23.767 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T12:37:23.792 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T12:37:23.857 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:37:23.858 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T12:37:23.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T12:37:23.953 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-28T12:37:24.072 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-28T12:37:24.073 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:37:24.073 INFO:tasks.cephadm:Cluster fsid is 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:37:24.073 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-28T12:37:24.074 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.53', 'mon.b': '172.21.15.77', 'mon.c': '172.21.15.191'} 2024-04-28T12:37:24.074 INFO:tasks.cephadm:First mon is mon.a on smithi053 2024-04-28T12:37:24.074 INFO:tasks.cephadm:First mgr is x 2024-04-28T12:37:24.074 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-28T12:37:24.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-04-28T12:37:24.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-04-28T12:37:24.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-04-28T12:37:24.154 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-28T12:37:24.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:37:24.290 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-28T12:37:24.513 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:37:24.515 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-28T12:37:24.515 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:25.272 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 12:37 /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:25.273 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:26.013 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 12:37 /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:26.014 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:26.507 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 12:37 /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:26.507 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:26.528 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:26.547 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T12:37:26.582 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-28T12:37:26.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T12:37:26.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T12:37:26.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T12:37:28.046 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:37:28.164 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:37:28.360 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:38:11.820 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-28T12:38:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T12:38:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T12:38:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-04-28T12:38:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T12:38:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-04-28T12:38:11.821 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-28T12:40:01.392 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-28T12:40:01.393 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T12:40:01.393 INFO:teuthology.orchestra.run.smithi053.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T12:40:01.393 INFO:teuthology.orchestra.run.smithi053.stdout: "repo_digests": [ 2024-04-28T12:40:01.393 INFO:teuthology.orchestra.run.smithi053.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T12:40:01.393 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-04-28T12:40:01.393 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-28T12:40:19.896 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-04-28T12:40:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T12:40:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T12:40:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-04-28T12:40:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T12:40:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-04-28T12:40:19.897 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-04-28T12:40:19.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-04-28T12:40:19.955 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-04-28T12:40:19.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-04-28T12:40:20.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-04-28T12:40:20.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-04-28T12:40:20.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-04-28T12:40:20.094 INFO:tasks.cephadm:Writing seed config... 2024-04-28T12:40:20.095 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-28T12:40:20.095 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-28T12:40:20.095 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-28T12:40:20.095 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-28T12:40:20.095 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-28T12:40:20.096 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-28T12:40:20.097 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-28T12:40:20.098 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-28T12:40:20.099 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-28T12:40:20.100 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:40:20.100 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-28T12:40:20.119 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 10525726-055c-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-28T12:40:20.119 DEBUG:teuthology.orchestra.run.smithi053:mon.a> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a.service 2024-04-28T12:40:20.162 DEBUG:teuthology.orchestra.run.smithi053:mgr.x> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.x.service 2024-04-28T12:40:20.205 INFO:tasks.cephadm:Bootstrapping... 2024-04-28T12:40:20.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid 10525726-055c-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-28T12:40:20.409 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:40:20.409 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', '10525726-055c-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.53', '--skip-admin-label'] 2024-04-28T12:40:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-28T12:40:20.410 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying podman|docker is present... 2024-04-28T12:40:20.447 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-28T12:40:20.447 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying lvm2 is present... 2024-04-28T12:40:20.447 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying time synchronization is in place... 2024-04-28T12:40:20.455 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-28T12:40:20.455 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-28T12:40:20.461 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-28T12:40:20.461 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-04-28T12:40:20.467 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout enabled 2024-04-28T12:40:20.474 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout active 2024-04-28T12:40:20.474 INFO:teuthology.orchestra.run.smithi053.stdout:Unit chronyd.service is enabled and running 2024-04-28T12:40:20.474 INFO:teuthology.orchestra.run.smithi053.stdout:Repeating the final host check... 2024-04-28T12:40:20.498 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-28T12:40:20.498 INFO:teuthology.orchestra.run.smithi053.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-28T12:40:20.499 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl is present 2024-04-28T12:40:20.499 INFO:teuthology.orchestra.run.smithi053.stdout:lvcreate is present 2024-04-28T12:40:20.505 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-28T12:40:20.505 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-28T12:40:20.511 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-28T12:40:20.511 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-04-28T12:40:20.517 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout enabled 2024-04-28T12:40:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout active 2024-04-28T12:40:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:Unit chronyd.service is enabled and running 2024-04-28T12:40:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:Host looks OK 2024-04-28T12:40:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:Cluster fsid: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:Acquiring lock 140677696954320 on /run/cephadm/10525726-055c-11ef-bc93-c7b262605968.lock 2024-04-28T12:40:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:Lock 140677696954320 acquired on /run/cephadm/10525726-055c-11ef-bc93-c7b262605968.lock 2024-04-28T12:40:20.525 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying IP 172.21.15.53 port 3300 ... 2024-04-28T12:40:20.525 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying IP 172.21.15.53 port 6789 ... 2024-04-28T12:40:20.525 INFO:teuthology.orchestra.run.smithi053.stdout:Base mon IP(s) is [172.21.15.53:3300, 172.21.15.53:6789], mon addrv is [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-04-28T12:40:20.528 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.53 metric 100 2024-04-28T12:40:20.528 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.53 metric 100 2024-04-28T12:40:20.531 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-28T12:40:20.531 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-28T12:40:20.531 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-28T12:40:20.533 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-28T12:40:20.533 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-28T12:40:20.533 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-28T12:40:20.533 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-28T12:40:20.534 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-04-28T12:40:20.534 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-28T12:40:20.534 INFO:teuthology.orchestra.run.smithi053.stdout:Mon IP `172.21.15.53` is in CIDR network `172.21.0.0/20` 2024-04-28T12:40:20.534 INFO:teuthology.orchestra.run.smithi053.stdout:Mon IP `172.21.15.53` is in CIDR network `172.21.0.0/20` 2024-04-28T12:40:20.534 INFO:teuthology.orchestra.run.smithi053.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-28T12:40:20.535 INFO:teuthology.orchestra.run.smithi053.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-28T12:40:20.535 INFO:teuthology.orchestra.run.smithi053.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:40:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-28T12:40:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:40:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Getting image source signatures 2024-04-28T12:40:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-28T12:40:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-28T12:40:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-28T12:40:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-28T12:40:21.920 INFO:teuthology.orchestra.run.smithi053.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-28T12:40:21.920 INFO:teuthology.orchestra.run.smithi053.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-28T12:40:21.920 INFO:teuthology.orchestra.run.smithi053.stdout:Extracting ceph user uid/gid from container image... 2024-04-28T12:40:28.552 INFO:teuthology.orchestra.run.smithi053.stdout:stat: stdout 167 167 2024-04-28T12:40:28.552 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial keys... 2024-04-28T12:40:32.570 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQC/Qy5mwfAMChAAGO+Otekt6iIlXsSLOuGphw== 2024-04-28T12:40:33.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQDAQy5mHTAoNRAAK2bob99BgZ/BkPvANbOoLg== 2024-04-28T12:40:34.677 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQDBQy5mPrlWKRAAA4h54in9m//DY8RgmM2gIw== 2024-04-28T12:40:34.678 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial monmap... 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool for a [v2:172.21.15.53:3300,v1:172.21.15.53:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:setting min_mon_release = quincy 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: set fsid to 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-28T12:40:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:40:35.610 INFO:teuthology.orchestra.run.smithi053.stdout:Creating mon... 2024-04-28T12:40:36.514 INFO:teuthology.orchestra.run.smithi053.stdout:create mon.a on 2024-04-28T12:40:36.746 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-28T12:40:36.891 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-28T12:40:37.073 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-10525726-055c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-10525726-055c-11ef-bc93-c7b262605968.target. 2024-04-28T12:40:37.073 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-10525726-055c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-10525726-055c-11ef-bc93-c7b262605968.target. 2024-04-28T12:40:37.911 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a 2024-04-28T12:40:37.911 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to reset failed state of unit ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-28T12:40:38.194 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-10525726-055c-11ef-bc93-c7b262605968.target.wants/ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-10525726-055c-11ef-bc93-c7b262605968@.service. 2024-04-28T12:40:38.828 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:38 smithi053 podman[28795]: 2024-04-28 12:40:38.603980621 +0000 UTC m=+0.018771115 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:40:38.828 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:38 smithi053 podman[28795]: 2024-04-28 12:40:38.832051154 +0000 UTC m=+0.246841637 container create b799ed646dc04f7b3be8bc5f3b25a818d2f700f8bd9c3658f0d5defb739cac56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-28T12:40:39.196 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:40:39.196 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-04-28T12:40:39.203 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:40:39.203 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-04-28T12:40:39.204 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-04-28T12:40:39.204 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to enable service . firewalld.service is not available 2024-04-28T12:40:39.204 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mon to start... 2024-04-28T12:40:39.204 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mon... 2024-04-28T12:40:39.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:39 smithi053 podman[28795]: 2024-04-28 12:40:39.094494456 +0000 UTC m=+0.509284936 container init b799ed646dc04f7b3be8bc5f3b25a818d2f700f8bd9c3658f0d5defb739cac56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-28T12:40:39.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:39 smithi053 podman[28795]: 2024-04-28 12:40:39.097660306 +0000 UTC m=+0.512450786 container start b799ed646dc04f7b3be8bc5f3b25a818d2f700f8bd9c3658f0d5defb739cac56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:40:39.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:39 smithi053 ceph-mon[28815]: mkfs 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:39.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:39 smithi053 ceph-mon[28815]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T12:40:39.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:39 smithi053 bash[28795]: b799ed646dc04f7b3be8bc5f3b25a818d2f700f8bd9c3658f0d5defb739cac56 2024-04-28T12:40:39.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:39 smithi053 systemd[1]: Started Ceph mon.a for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:40:40.389 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T12:40:40.389 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: monmap epoch 1 2024-04-28T12:40:40.390 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:40.390 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: last_changed 2024-04-28T12:40:35.008027+0000 2024-04-28T12:40:40.390 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:40:40.390 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: min_mon_release 19 (squid) 2024-04-28T12:40:40.390 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: election_strategy: 1 2024-04-28T12:40:40.390 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:40:40.391 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: fsmap 2024-04-28T12:40:40.391 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: osdmap e1: 0 total, 0 up, 0 in 2024-04-28T12:40:40.391 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: mgrmap e1: no daemons active 2024-04-28T12:40:40.391 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:40 smithi053 ceph-mon[28815]: from='client.? 172.21.15.53:0/1421135630' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-28T12:40:40.392 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout cluster: 2024-04-28T12:40:40.400 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout id: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:40.400 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-28T12:40:40.400 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:40.400 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout services: 2024-04-28T12:40:40.400 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.527123s) 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout data: 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout pgs: 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.582+0000 7fcd85c1a700 1 Processor -- start 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.582+0000 7fcd85c1a700 1 -- start start 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.582+0000 7fcd85c1a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd801091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.582+0000 7fcd85c1a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcd801097c0 con 0x7fcd80108dd0 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd801091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd801091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46730/0 (socket says 172.21.15.53:46730) 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7f7fe700 1 -- 172.21.15.53:0/1894293873 learned_addr learned my addr 172.21.15.53:0/1894293873 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7f7fe700 1 -- 172.21.15.53:0/1894293873 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd80109f80 con 0x7fcd80108dd0 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7f7fe700 1 --2- 172.21.15.53:0/1894293873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd801091f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009e80 tx=0x7fcd7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbb643252717f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7e7fc700 1 -- 172.21.15.53:0/1894293873 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd70017070 con 0x7fcd80108dd0 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7e7fc700 1 -- 172.21.15.53:0/1894293873 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70003a20 con 0x7fcd80108dd0 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.583+0000 7fcd7e7fc700 1 -- 172.21.15.53:0/1894293873 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd70003d20 con 0x7fcd80108dd0 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1894293873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 msgr2=0x7fcd801091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 --2- 172.21.15.53:0/1894293873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd801091f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009e80 tx=0x7fcd7000b330 comp rx=0 tx=0).stop 2024-04-28T12:40:40.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1894293873 shutdown_connections 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 --2- 172.21.15.53:0/1894293873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd801091f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1894293873 >> 172.21.15.53:0/1894293873 conn(0x7fcd80104370 msgr2=0x7fcd801067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1894293873 shutdown_connections 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1894293873 wait complete. 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.584+0000 7fcd85c1a700 1 Processor -- start 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd85c1a700 1 -- start start 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd85c1a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd8007f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd85c1a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8010a4b0 con 0x7fcd80108dd0 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd7f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd8007f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd7f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd8007f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46732/0 (socket says 172.21.15.53:46732) 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd7f7fe700 1 -- 172.21.15.53:0/1421135630 learned_addr learned my addr 172.21.15.53:0/1421135630 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd7f7fe700 1 -- 172.21.15.53:0/1421135630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8007f930 con 0x7fcd80108dd0 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.585+0000 7fcd7f7fe700 1 --2- 172.21.15.53:0/1421135630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd8007f3f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcd70004000 tx=0x7fcd70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:40.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.586+0000 7fcd7cff9700 1 -- 172.21.15.53:0/1421135630 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd70022020 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.586+0000 7fcd7cff9700 1 -- 172.21.15.53:0/1421135630 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70018040 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.586+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8007db20 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.586+0000 7fcd7cff9700 1 -- 172.21.15.53:0/1421135630 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd70017070 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.586+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd8007e060 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.586+0000 7fcd7cff9700 1 -- 172.21.15.53:0/1421135630 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcd70025020 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.586+0000 7fcd7cff9700 1 -- 172.21.15.53:0/1421135630 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcd7000f050 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.587+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd80064a80 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.589+0000 7fcd7cff9700 1 -- 172.21.15.53:0/1421135630 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fcd7002c030 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.662+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fcd80195a20 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.663+0000 7fcd7cff9700 1 -- 172.21.15.53:0/1421135630 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fcd70012810 con 0x7fcd80108dd0 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.663+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 msgr2=0x7fcd8007f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.663+0000 7fcd85c1a700 1 --2- 172.21.15.53:0/1421135630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd8007f3f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcd70004000 tx=0x7fcd70004ad0 comp rx=0 tx=0).stop 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.663+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 shutdown_connections 2024-04-28T12:40:40.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.663+0000 7fcd85c1a700 1 --2- 172.21.15.53:0/1421135630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd80108dd0 0x7fcd8007f3f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:40.405 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.663+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 >> 172.21.15.53:0/1421135630 conn(0x7fcd80104370 msgr2=0x7fcd80193180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:40.405 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.663+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 shutdown_connections 2024-04-28T12:40:40.405 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:39.664+0000 7fcd85c1a700 1 -- 172.21.15.53:0/1421135630 wait complete. 2024-04-28T12:40:40.405 INFO:teuthology.orchestra.run.smithi053.stdout:mon is available 2024-04-28T12:40:40.405 INFO:teuthology.orchestra.run.smithi053.stdout:Assimilating anything we can from ceph.conf... 2024-04-28T12:40:41.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:41 smithi053 ceph-mon[28815]: from='client.? 172.21.15.53:0/3929911403' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-28T12:40:41.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:41 smithi053 ceph-mon[28815]: from='client.? 172.21.15.53:0/3929911403' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [global] 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout fsid = 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-28T12:40:41.452 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [osd] 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.800+0000 7fc0216d1700 1 Processor -- start 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.800+0000 7fc0216d1700 1 -- start start 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.800+0000 7fc0216d1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.800+0000 7fc0216d1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c1097c0 con 0x7fc01c108dd0 2024-04-28T12:40:41.453 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.800+0000 7fc01affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.800+0000 7fc01affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46748/0 (socket says 172.21.15.53:46748) 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.800+0000 7fc01affd700 1 -- 172.21.15.53:0/2436286833 learned_addr learned my addr 172.21.15.53:0/2436286833 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.801+0000 7fc01affd700 1 -- 172.21.15.53:0/2436286833 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c109f80 con 0x7fc01c108dd0 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.801+0000 7fc01affd700 1 --2- 172.21.15.53:0/2436286833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c1091f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009e80 tx=0x7fc00c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d465b7def281add1 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.801+0000 7fc019ffb700 1 -- 172.21.15.53:0/2436286833 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc00c017070 con 0x7fc01c108dd0 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.801+0000 7fc019ffb700 1 -- 172.21.15.53:0/2436286833 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00c003a20 con 0x7fc01c108dd0 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.801+0000 7fc019ffb700 1 -- 172.21.15.53:0/2436286833 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc00c003d20 con 0x7fc01c108dd0 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.801+0000 7fc0216d1700 1 -- 172.21.15.53:0/2436286833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 msgr2=0x7fc01c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.801+0000 7fc0216d1700 1 --2- 172.21.15.53:0/2436286833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c1091f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009e80 tx=0x7fc00c00b330 comp rx=0 tx=0).stop 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 -- 172.21.15.53:0/2436286833 shutdown_connections 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 --2- 172.21.15.53:0/2436286833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c1091f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 -- 172.21.15.53:0/2436286833 >> 172.21.15.53:0/2436286833 conn(0x7fc01c104370 msgr2=0x7fc01c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:41.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 -- 172.21.15.53:0/2436286833 shutdown_connections 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 -- 172.21.15.53:0/2436286833 wait complete. 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 Processor -- start 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 -- start start 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.802+0000 7fc0216d1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc0216d1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c10a4b0 con 0x7fc01c108dd0 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc01affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc01affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46756/0 (socket says 172.21.15.53:46756) 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc01affd700 1 -- 172.21.15.53:0/3929911403 learned_addr learned my addr 172.21.15.53:0/3929911403 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc01affd700 1 -- 172.21.15.53:0/3929911403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c19ca50 con 0x7fc01c108dd0 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc01affd700 1 --2- 172.21.15.53:0/3929911403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c19c510 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc00c004000 tx=0x7fc00c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc00c022020 con 0x7fc01c108dd0 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00c018040 con 0x7fc01c108dd0 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01c19cce0 con 0x7fc01c108dd0 2024-04-28T12:40:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc00c017070 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01c19f9d0 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc00c025020 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.803+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc00c00f050 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.804+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc01c064a80 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.807+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fc00c02c030 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.877+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc01c02e620 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.880+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fc00c01e480 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.881+0000 7fc013fff700 1 -- 172.21.15.53:0/3929911403 <== mon.0 v2:172.21.15.53:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc00c003a20 con 0x7fc01c108dd0 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.881+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 msgr2=0x7fc01c19c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.881+0000 7fc0216d1700 1 --2- 172.21.15.53:0/3929911403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c19c510 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc00c004000 tx=0x7fc00c004ad0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.881+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 shutdown_connections 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.881+0000 7fc0216d1700 1 --2- 172.21.15.53:0/3929911403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc01c108dd0 0x7fc01c19c510 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.456 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.881+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 >> 172.21.15.53:0/3929911403 conn(0x7fc01c104370 msgr2=0x7fc01c193180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:41.457 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.882+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 shutdown_connections 2024-04-28T12:40:41.457 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.882+0000 7fc0216d1700 1 -- 172.21.15.53:0/3929911403 wait complete. 2024-04-28T12:40:41.457 INFO:teuthology.orchestra.run.smithi053.stdout:Generating new minimal ceph.conf... 2024-04-28T12:40:42.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:42 smithi053 ceph-mon[28815]: from='client.? 172.21.15.53:0/292961699' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:40:42.495 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.826+0000 7fd473fc8700 1 Processor -- start 2024-04-28T12:40:42.495 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.826+0000 7fd473fc8700 1 -- start start 2024-04-28T12:40:42.495 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.826+0000 7fd473fc8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:42.495 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.826+0000 7fd473fc8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1097c0 con 0x7fd46c108dd0 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.826+0000 7fd471d64700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.826+0000 7fd471d64700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46758/0 (socket says 172.21.15.53:46758) 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.826+0000 7fd471d64700 1 -- 172.21.15.53:0/996330508 learned_addr learned my addr 172.21.15.53:0/996330508 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.827+0000 7fd471d64700 1 -- 172.21.15.53:0/996330508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46c109f80 con 0x7fd46c108dd0 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.827+0000 7fd471d64700 1 --2- 172.21.15.53:0/996330508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c1091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd4680098a0 tx=0x7fd468009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bbf2eadda33773 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.827+0000 7fd470d62700 1 -- 172.21.15.53:0/996330508 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd468004160 con 0x7fd46c108dd0 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.827+0000 7fd470d62700 1 -- 172.21.15.53:0/996330508 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd468004300 con 0x7fd46c108dd0 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.827+0000 7fd470d62700 1 -- 172.21.15.53:0/996330508 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd468011af0 con 0x7fd46c108dd0 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.827+0000 7fd473fc8700 1 -- 172.21.15.53:0/996330508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 msgr2=0x7fd46c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.827+0000 7fd473fc8700 1 --2- 172.21.15.53:0/996330508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c1091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd4680098a0 tx=0x7fd468009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:42.496 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 -- 172.21.15.53:0/996330508 shutdown_connections 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 --2- 172.21.15.53:0/996330508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c1091f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 -- 172.21.15.53:0/996330508 >> 172.21.15.53:0/996330508 conn(0x7fd46c104370 msgr2=0x7fd46c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 -- 172.21.15.53:0/996330508 shutdown_connections 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 -- 172.21.15.53:0/996330508 wait complete. 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 Processor -- start 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 -- start start 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.828+0000 7fd473fc8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c10d330 con 0x7fd46c108dd0 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd471d64700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd471d64700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c19cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46774/0 (socket says 172.21.15.53:46774) 2024-04-28T12:40:42.497 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd471d64700 1 -- 172.21.15.53:0/292961699 learned_addr learned my addr 172.21.15.53:0/292961699 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd471d64700 1 -- 172.21.15.53:0/292961699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46c19d250 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd471d64700 1 --2- 172.21.15.53:0/292961699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c19cd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd468009fb0 tx=0x7fd468011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd462ffd700 1 -- 172.21.15.53:0/292961699 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4680036a0 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd46c19d4e0 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd462ffd700 1 -- 172.21.15.53:0/292961699 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd46801a430 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd462ffd700 1 -- 172.21.15.53:0/292961699 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd468019550 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46c1a01c0 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.829+0000 7fd462ffd700 1 -- 172.21.15.53:0/292961699 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd468019be0 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.830+0000 7fd462ffd700 1 -- 172.21.15.53:0/292961699 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd468024730 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.830+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd46c064a80 con 0x7fd46c108dd0 2024-04-28T12:40:42.498 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.833+0000 7fd462ffd700 1 -- 172.21.15.53:0/292961699 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd468043ec0 con 0x7fd46c108dd0 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.903+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd46c1a0390 con 0x7fd46c108dd0 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.904+0000 7fd462ffd700 1 -- 172.21.15.53:0/292961699 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fd468032020 con 0x7fd46c108dd0 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.904+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 msgr2=0x7fd46c19cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.904+0000 7fd473fc8700 1 --2- 172.21.15.53:0/292961699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c19cd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd468009fb0 tx=0x7fd468011940 comp rx=0 tx=0).stop 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.904+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 shutdown_connections 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.904+0000 7fd473fc8700 1 --2- 172.21.15.53:0/292961699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd46c108dd0 0x7fd46c19cd10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.904+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 >> 172.21.15.53:0/292961699 conn(0x7fd46c104370 msgr2=0x7fd46c104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.905+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 shutdown_connections 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:41.905+0000 7fd473fc8700 1 -- 172.21.15.53:0/292961699 wait complete. 2024-04-28T12:40:42.499 INFO:teuthology.orchestra.run.smithi053.stdout:Restarting the monitor... 2024-04-28T12:40:42.783 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:42 smithi053 systemd[1]: Stopping Ceph mon.a for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:40:43.048 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:42 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a[28811]: 2024-04-28T12:40:42.785+0000 7f51b2355700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T12:40:43.049 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:42 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a[28811]: 2024-04-28T12:40:42.785+0000 7f51b2355700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-28T12:40:43.049 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:42 smithi053 podman[29058]: 2024-04-28 12:40:42.848187672 +0000 UTC m=+0.179681271 container died b799ed646dc04f7b3be8bc5f3b25a818d2f700f8bd9c3658f0d5defb739cac56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-28T12:40:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:43 smithi053 podman[29058]: 2024-04-28 12:40:43.127857168 +0000 UTC m=+0.459350772 container cleanup b799ed646dc04f7b3be8bc5f3b25a818d2f700f8bd9c3658f0d5defb739cac56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-28T12:40:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:43 smithi053 bash[29058]: ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a 2024-04-28T12:40:44.109 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:43 smithi053 podman[29073]: 2024-04-28 12:40:43.737324795 +0000 UTC m=+0.881081732 container remove b799ed646dc04f7b3be8bc5f3b25a818d2f700f8bd9c3658f0d5defb739cac56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:40:44.109 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:43 smithi053 systemd[1]: ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-28T12:40:44.109 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:43 smithi053 systemd[1]: Stopped Ceph mon.a for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:40:44.109 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:43 smithi053 systemd[1]: Starting Ceph mon.a for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:40:44.394 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 podman[29163]: 2024-04-28 12:40:44.113076538 +0000 UTC m=+0.089588493 container create bd4ec601fb045968ff6b3b6e181d0d2f3baf1dfddc183383b0ec6218c7422e36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 podman[29163]: 2024-04-28 12:40:44.042123996 +0000 UTC m=+0.018635970 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 podman[29163]: 2024-04-28 12:40:44.323796042 +0000 UTC m=+0.300307997 container init bd4ec601fb045968ff6b3b6e181d0d2f3baf1dfddc183383b0ec6218c7422e36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 podman[29163]: 2024-04-28 12:40:44.326822008 +0000 UTC m=+0.303333964 container start bd4ec601fb045968ff6b3b6e181d0d2f3baf1dfddc183383b0ec6218c7422e36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: set uid:gid to 167:167 (ceph:ceph) 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: pidfile_write: ignore empty --pid-file 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: load: jerasure load: lrc 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: RocksDB version: 7.9.2 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Git sha 0 2024-04-28T12:40:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-28T12:40:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: DB SUMMARY 2024-04-28T12:40:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: DB Session ID: WM6DQ40EH6XRMTFJ98GA 2024-04-28T12:40:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: CURRENT file: CURRENT 2024-04-28T12:40:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: IDENTITY file: IDENTITY 2024-04-28T12:40:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-28T12:40:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-28T12:40:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104650 ; 2024-04-28T12:40:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.error_if_exists: 0 2024-04-28T12:40:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.create_if_missing: 0 2024-04-28T12:40:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.paranoid_checks: 1 2024-04-28T12:40:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-28T12:40:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-28T12:40:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-28T12:40:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.env: 0x55ed15c81cc0 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.fs: PosixFileSystem 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.info_log: 0x55ed1655bd80 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_file_opening_threads: 16 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.statistics: (nil) 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.use_fsync: 0 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_log_file_size: 0 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.keep_log_file_num: 1000 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.recycle_log_file_num: 0 2024-04-28T12:40:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.allow_fallocate: 1 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.allow_mmap_reads: 0 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.allow_mmap_writes: 0 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.use_direct_reads: 0 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.create_missing_column_families: 0 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.db_log_dir: 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.wal_dir: 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-28T12:40:44.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.advise_random_on_open: 1 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.db_write_buffer_size: 0 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.write_buffer_manager: 0x55ed1652d0e0 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.rate_limiter: (nil) 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.wal_recovery_mode: 2 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.enable_thread_tracking: 0 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.enable_pipelined_write: 0 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.unordered_write: 0 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-28T12:40:44.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.row_cache: None 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.wal_filter: None 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.allow_ingest_behind: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.two_write_queues: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.manual_wal_flush: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.wal_compression: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.atomic_flush: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-28T12:40:44.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.log_readahead_size: 0 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.best_efforts_recovery: 0 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.allow_data_in_errors: 0 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.db_host_id: __hostname__ 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_background_jobs: 2 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_background_compactions: -1 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_subcompactions: 1 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-28T12:40:44.402 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_total_wal_size: 0 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_open_files: -1 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bytes_per_sync: 0 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_readahead_size: 0 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_background_flushes: -1 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Compression algorithms supported: 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kZSTD supported: 0 2024-04-28T12:40:44.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kXpressCompression supported: 0 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kLZ4HCCompression supported: 1 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kZlibCompression supported: 1 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kSnappyCompression supported: 1 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kLZ4Compression supported: 1 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: kBZip2Compression supported: 0 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.merge_operator: 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_filter: None 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_filter_factory: None 2024-04-28T12:40:44.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.sst_partitioner_factory: None 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ed1655bf40) 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_type: 0 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: data_block_index_type: 0 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_shortening: 1 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: checksum: 4 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: no_block_cache: 0 2024-04-28T12:40:44.405 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache: 0x55ed1655f1f0 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_options: 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: capacity : 536870912 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: num_shard_bits : 4 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: strict_capacity_limit : 0 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_compressed: (nil) 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: persistent_cache: (nil) 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_size: 4096 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_size_deviation: 10 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_restart_interval: 16 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_block_restart_interval: 1 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: metadata_block_size: 4096 2024-04-28T12:40:44.406 INFO:journalctl@ceph.mon.a.smithi053.stdout: partition_filters: 0 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: use_delta_encoding: 1 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: filter_policy: bloomfilter 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: whole_key_filtering: 1 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: verify_compression: 0 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: format_version: 5 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: enable_index_compression: 1 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_align: 0 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: max_auto_readahead_size: 262144 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: prepopulate_block_cache: 0 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: initial_auto_readahead_size: 8192 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout: num_file_reads_for_auto_readahead: 2 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.write_buffer_size: 33554432 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_write_buffer_number: 2 2024-04-28T12:40:44.407 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression: NoCompression 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression: Disabled 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.prefix_extractor: nullptr 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.num_levels: 7 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-28T12:40:44.408 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.level: 32767 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.strategy: 0 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.enabled: false 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-28T12:40:44.409 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.target_file_size_base: 67108864 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-28T12:40:44.410 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.arena_block_size: 1048576 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-28T12:40:44.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.disable_auto_compactions: 0 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-28T12:40:44.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-28T12:40:44.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-28T12:40:44.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-28T12:40:44.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-28T12:40:44.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.inplace_update_support: 0 2024-04-28T12:40:44.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-28T12:40:44.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.bloom_locality: 0 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.max_successive_merges: 0 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.paranoid_file_checks: 0 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.force_consistency_checks: 1 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.report_bg_io_stats: 0 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.ttl: 2592000 2024-04-28T12:40:44.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.enable_blob_files: false 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.min_blob_size: 0 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.blob_file_size: 268435456 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.blob_file_starting_level: 0 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-28T12:40:44.416 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 301263ae-f467-40e2-89c0-7505e9ec898e 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308044355828, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308044356941, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99850, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97736, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14058, "raw_average_key_size": 49, "raw_value_size": 90087, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714308044, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "301263ae-f467-40e2-89c0-7505e9ec898e", "db_session_id": "WM6DQ40EH6XRMTFJ98GA", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308044357036, "job": 1, "event": "recovery_finished"} 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ed1667c000 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: DB pointer 0x55ed16668000 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** DB Stats ** 2024-04-28T12:40:44.417 INFO:journalctl@ceph.mon.a.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: L0 2/0 99.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Sum 2/0 99.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:40:44.418 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-28T12:40:44.419 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative compaction: 0.00 GB write, 30.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval compaction: 0.00 GB write, 30.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: Block cache BinnedLRUCache@0x55ed1655f1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: starting mon.a rank 0 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???) e1 preinit fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???).mds e1 new map 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???).mds e1 print_map 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: e1 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T12:40:44.420 INFO:journalctl@ceph.mon.a.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout: legacy client fscid: -1 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout: No filesystems configured 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 bash[29163]: bd4ec601fb045968ff6b3b6e181d0d2f3baf1dfddc183383b0ec6218c7422e36 2024-04-28T12:40:44.421 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 systemd[1]: Started Ceph mon.a for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:40:44.421 INFO:teuthology.orchestra.run.smithi053.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-28T12:40:44.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T12:40:44.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: monmap epoch 1 2024-04-28T12:40:44.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:44.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: last_changed 2024-04-28T12:40:35.008027+0000 2024-04-28T12:40:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:40:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: min_mon_release 19 (squid) 2024-04-28T12:40:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: election_strategy: 1 2024-04-28T12:40:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:40:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: fsmap 2024-04-28T12:40:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: osdmap e1: 0 total, 0 up, 0 in 2024-04-28T12:40:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:44 smithi053 ceph-mon[29182]: mgrmap e1: no daemons active 2024-04-28T12:40:45.454 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.848+0000 7f7f186d4700 1 Processor -- start 2024-04-28T12:40:45.458 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.848+0000 7f7f186d4700 1 -- start start 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.848+0000 7f7f186d4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1010bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.848+0000 7f7f186d4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1007d690 con 0x7f7f10109880 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f16470700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1010bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f16470700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1010bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46778/0 (socket says 172.21.15.53:46778) 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f16470700 1 -- 172.21.15.53:0/2244642438 learned_addr learned my addr 172.21.15.53:0/2244642438 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f16470700 1 -- 172.21.15.53:0/2244642438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1010c1b0 con 0x7f7f10109880 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f16470700 1 --2- 172.21.15.53:0/2244642438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1010bc70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00f960 tx=0x7f7f0c00fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76449beb4bd8c0d server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f1546e700 1 -- 172.21.15.53:0/2244642438 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f0c01c070 con 0x7f7f10109880 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f1546e700 1 -- 172.21.15.53:0/2244642438 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7f0c004030 con 0x7f7f10109880 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.849+0000 7f7f1546e700 1 -- 172.21.15.53:0/2244642438 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f0c003c80 con 0x7f7f10109880 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.850+0000 7f7f186d4700 1 -- 172.21.15.53:0/2244642438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 msgr2=0x7f7f1010bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.850+0000 7f7f186d4700 1 --2- 172.21.15.53:0/2244642438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1010bc70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00f960 tx=0x7f7f0c00fc70 comp rx=0 tx=0).stop 2024-04-28T12:40:45.459 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.850+0000 7f7f186d4700 1 -- 172.21.15.53:0/2244642438 shutdown_connections 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.850+0000 7f7f186d4700 1 --2- 172.21.15.53:0/2244642438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1010bc70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.850+0000 7f7f186d4700 1 -- 172.21.15.53:0/2244642438 >> 172.21.15.53:0/2244642438 conn(0x7f7f10103230 msgr2=0x7f7f10105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.850+0000 7f7f186d4700 1 -- 172.21.15.53:0/2244642438 shutdown_connections 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.850+0000 7f7f186d4700 1 -- 172.21.15.53:0/2244642438 wait complete. 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f186d4700 1 Processor -- start 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f186d4700 1 -- start start 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f186d4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f186d4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7f10108400 con 0x7f7f10109880 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f16470700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f16470700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1019aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46794/0 (socket says 172.21.15.53:46794) 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f16470700 1 -- 172.21.15.53:0/571179874 learned_addr learned my addr 172.21.15.53:0/571179874 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f16470700 1 -- 172.21.15.53:0/571179874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1019aff0 con 0x7f7f10109880 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.851+0000 7f7f16470700 1 --2- 172.21.15.53:0/571179874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1019aab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c000c00 tx=0x7f7f0c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.852+0000 7f7f077fe700 1 -- 172.21.15.53:0/571179874 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f0c01c040 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.852+0000 7f7f077fe700 1 -- 172.21.15.53:0/571179874 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7f0c025070 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.852+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1019b280 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.852+0000 7f7f077fe700 1 -- 172.21.15.53:0/571179874 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f0c020610 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.852+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f1019df70 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.852+0000 7f7f077fe700 1 -- 172.21.15.53:0/571179874 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f0c020850 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.852+0000 7f7f077fe700 1 -- 172.21.15.53:0/571179874 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7f0c014070 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.853+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f10064a80 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.855+0000 7f7f077fe700 1 -- 172.21.15.53:0/571179874 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f7f0c020cd0 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.926+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f7f1019b700 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.928+0000 7f7f077fe700 1 -- 172.21.15.53:0/571179874 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f7f0c017a00 con 0x7f7f10109880 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.929+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 msgr2=0x7f7f1019aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.929+0000 7f7f186d4700 1 --2- 172.21.15.53:0/571179874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1019aab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c000c00 tx=0x7f7f0c0044c0 comp rx=0 tx=0).stop 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.929+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 shutdown_connections 2024-04-28T12:40:45.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.929+0000 7f7f186d4700 1 --2- 172.21.15.53:0/571179874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f10109880 0x7f7f1019aab0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:45.462 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.929+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 >> 172.21.15.53:0/571179874 conn(0x7f7f10103230 msgr2=0x7f7f10103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:45.462 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.929+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 shutdown_connections 2024-04-28T12:40:45.462 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:44.929+0000 7f7f186d4700 1 -- 172.21.15.53:0/571179874 wait complete. 2024-04-28T12:40:45.462 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-28T12:40:45.475 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-28T12:40:45.475 INFO:teuthology.orchestra.run.smithi053.stdout:Creating mgr... 2024-04-28T12:40:45.476 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-28T12:40:45.476 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-28T12:40:45.715 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.x 2024-04-28T12:40:45.715 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to reset failed state of unit ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-28T12:40:45.870 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-10525726-055c-11ef-bc93-c7b262605968.target.wants/ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-10525726-055c-11ef-bc93-c7b262605968@.service. 2024-04-28T12:40:46.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/571179874' entity='client.admin' 2024-04-28T12:40:46.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 podman[29403]: 2024-04-28 12:40:46.022926121 +0000 UTC m=+0.023163741 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:40:46.402 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:40:46.402 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-04-28T12:40:46.409 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:40:46.409 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-04-28T12:40:46.409 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-04-28T12:40:46.410 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to enable service . firewalld.service is not available 2024-04-28T12:40:46.418 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:40:46.418 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-04-28T12:40:46.425 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:40:46.426 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-04-28T12:40:46.426 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-04-28T12:40:46.426 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-28T12:40:46.426 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr to start... 2024-04-28T12:40:46.426 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr... 2024-04-28T12:40:46.477 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 podman[29403]: 2024-04-28 12:40:46.317538996 +0000 UTC m=+0.317776595 container init 271ec44b4c367648752f2c44c211aa50cfcb7ffcc6237b26387a044ddfcbc5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:40:46.477 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 podman[29403]: 2024-04-28 12:40:46.320030208 +0000 UTC m=+0.320267807 container start 271ec44b4c367648752f2c44c211aa50cfcb7ffcc6237b26387a044ddfcbc5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-28T12:40:46.477 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 bash[29403]: 271ec44b4c367648752f2c44c211aa50cfcb7ffcc6237b26387a044ddfcbc5f5 2024-04-28T12:40:46.477 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 systemd[1]: Started Ceph mgr.x for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:40:46.477 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:46.480+0000 7fb6fd3c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:40:46.779 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:46.673+0000 7fb6fd3c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:40:46.779 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:46.782+0000 7fb6fd3c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:40:47.344 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2544889139' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:40:47.345 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:47.122+0000 7fb6fd3c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:40:47.345 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:47.213+0000 7fb6fd3c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:40:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:40:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:40:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:40:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:40:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:40:47.649 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:40:47.650 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:40:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:40:47.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:40:47.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:40:47.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:47.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:47.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:47.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:40:47.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:40:39.138129+0000", 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.913+0000 7f8810f9a700 1 Processor -- start 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.913+0000 7f8810f9a700 1 -- start start 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.914+0000 7f8810f9a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c109680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.914+0000 7f8810f9a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f880c109c50 con 0x7f880c109260 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.914+0000 7f880a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c109680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.914+0000 7f880a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c109680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52338/0 (socket says 172.21.15.53:52338) 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.914+0000 7f880a59c700 1 -- 172.21.15.53:0/390496914 learned_addr learned my addr 172.21.15.53:0/390496914 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.915+0000 7f880a59c700 1 -- 172.21.15.53:0/390496914 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c107ec0 con 0x7f880c109260 2024-04-28T12:40:47.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.915+0000 7f880a59c700 1 --2- 172.21.15.53:0/390496914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c109680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f88000098a0 tx=0x7f8800009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792bd8747671f3b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.915+0000 7f880959a700 1 -- 172.21.15.53:0/390496914 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8800004160 con 0x7f880c109260 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.915+0000 7f880959a700 1 -- 172.21.15.53:0/390496914 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8800004300 con 0x7f880c109260 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.915+0000 7f880959a700 1 -- 172.21.15.53:0/390496914 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8800011af0 con 0x7f880c109260 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 -- 172.21.15.53:0/390496914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 msgr2=0x7f880c109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 --2- 172.21.15.53:0/390496914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c109680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f88000098a0 tx=0x7f8800009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 -- 172.21.15.53:0/390496914 shutdown_connections 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 --2- 172.21.15.53:0/390496914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c109680 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 -- 172.21.15.53:0/390496914 >> 172.21.15.53:0/390496914 conn(0x7f880c102970 msgr2=0x7f880c104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 -- 172.21.15.53:0/390496914 shutdown_connections 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 -- 172.21.15.53:0/390496914 wait complete. 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.916+0000 7f8810f9a700 1 Processor -- start 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f8810f9a700 1 -- start start 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f8810f9a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f8810f9a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f880c10c280 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f880a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f880a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c19ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52342/0 (socket says 172.21.15.53:52342) 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f880a59c700 1 -- 172.21.15.53:0/2544889139 learned_addr learned my addr 172.21.15.53:0/2544889139 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f880a59c700 1 -- 172.21.15.53:0/2544889139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c19f480 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f880a59c700 1 --2- 172.21.15.53:0/2544889139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c19ef40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8800009fb0 tx=0x7f8800011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f87fb7fe700 1 -- 172.21.15.53:0/2544889139 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88000036a0 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.917+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880c19f710 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.918+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f880c1a2400 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.918+0000 7f87fb7fe700 1 -- 172.21.15.53:0/2544889139 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f880001a430 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.918+0000 7f87fb7fe700 1 -- 172.21.15.53:0/2544889139 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8800019550 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.918+0000 7f87fb7fe700 1 -- 172.21.15.53:0/2544889139 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8800019b90 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.918+0000 7f87fb7fe700 1 -- 172.21.15.53:0/2544889139 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f880002dc80 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.919+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87ec005330 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.921+0000 7f87fb7fe700 1 -- 172.21.15.53:0/2544889139 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8800043ec0 con 0x7f880c109260 2024-04-28T12:40:47.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.994+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f87ec0058f0 con 0x7f880c109260 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.995+0000 7f87fb7fe700 1 -- 172.21.15.53:0/2544889139 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8800032020 con 0x7f880c109260 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.996+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 msgr2=0x7f880c19ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.996+0000 7f8810f9a700 1 --2- 172.21.15.53:0/2544889139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c19ef40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8800009fb0 tx=0x7f8800011940 comp rx=0 tx=0).stop 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.996+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 shutdown_connections 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.996+0000 7f8810f9a700 1 --2- 172.21.15.53:0/2544889139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c109260 0x7f880c19ef40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.996+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 >> 172.21.15.53:0/2544889139 conn(0x7f880c102970 msgr2=0x7f880c103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.997+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 shutdown_connections 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:46.997+0000 7f8810f9a700 1 -- 172.21.15.53:0/2544889139 wait complete. 2024-04-28T12:40:47.662 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (1/15)... 2024-04-28T12:40:48.867 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:48.502+0000 7fb6fd3c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:40:48.867 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:48.682+0000 7fb6fd3c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:40:48.867 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:48.870+0000 7fb6fd3c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:40:49.473 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:49.170+0000 7fb6fd3c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:40:49.474 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:49.267+0000 7fb6fd3c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:40:49.474 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:49.477+0000 7fb6fd3c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:40:49.762 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:49.677+0000 7fb6fd3c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:40:50.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:50 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4103939295' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:40:50.645 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:50.326+0000 7fb6fd3c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:50.747 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:40:50.748 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:40:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:50.750 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:40:39.138129+0000", 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.068+0000 7f64fa0ba700 1 Processor -- start 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.068+0000 7f64fa0ba700 1 -- start start 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.068+0000 7f64fa0ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f4107a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.068+0000 7f64fa0ba700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f64f407d680 con 0x7f64f4107650 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f4107a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f4107a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52356/0 (socket says 172.21.15.53:52356) 2024-04-28T12:40:50.751 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f37fe700 1 -- 172.21.15.53:0/386132387 learned_addr learned my addr 172.21.15.53:0/386132387 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f37fe700 1 -- 172.21.15.53:0/386132387 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f4107fb0 con 0x7f64f4107650 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f37fe700 1 --2- 172.21.15.53:0/386132387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f4107a70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64e80098a0 tx=0x7f64e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adfe40df66bc553 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f27fc700 1 -- 172.21.15.53:0/386132387 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e8004160 con 0x7f64f4107650 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f27fc700 1 -- 172.21.15.53:0/386132387 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f64e8004300 con 0x7f64f4107650 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.069+0000 7f64f27fc700 1 -- 172.21.15.53:0/386132387 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e8011af0 con 0x7f64f4107650 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.070+0000 7f64fa0ba700 1 -- 172.21.15.53:0/386132387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 msgr2=0x7f64f4107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.070+0000 7f64fa0ba700 1 --2- 172.21.15.53:0/386132387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f4107a70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64e80098a0 tx=0x7f64e8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.070+0000 7f64fa0ba700 1 -- 172.21.15.53:0/386132387 shutdown_connections 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.070+0000 7f64fa0ba700 1 --2- 172.21.15.53:0/386132387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f4107a70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.070+0000 7f64fa0ba700 1 -- 172.21.15.53:0/386132387 >> 172.21.15.53:0/386132387 conn(0x7f64f4103230 msgr2=0x7f64f4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.070+0000 7f64fa0ba700 1 -- 172.21.15.53:0/386132387 shutdown_connections 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.070+0000 7f64fa0ba700 1 -- 172.21.15.53:0/386132387 wait complete. 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64fa0ba700 1 Processor -- start 2024-04-28T12:40:50.752 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64fa0ba700 1 -- start start 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64fa0ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64fa0ba700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4108fe0 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64f37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64f37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f419eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52358/0 (socket says 172.21.15.53:52358) 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64f37fe700 1 -- 172.21.15.53:0/4103939295 learned_addr learned my addr 172.21.15.53:0/4103939295 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64f37fe700 1 -- 172.21.15.53:0/4103939295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f419f3e0 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.071+0000 7f64f37fe700 1 --2- 172.21.15.53:0/4103939295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f419eea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f64e8011ac0 tx=0x7f64e8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.072+0000 7f64f0ff9700 1 -- 172.21.15.53:0/4103939295 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e8003970 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.072+0000 7f64f0ff9700 1 -- 172.21.15.53:0/4103939295 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f64e8003b10 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.072+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f419f670 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.072+0000 7f64f0ff9700 1 -- 172.21.15.53:0/4103939295 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e8024610 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.072+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f41a2360 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.072+0000 7f64f0ff9700 1 -- 172.21.15.53:0/4103939295 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f64e8024c20 con 0x7f64f4107650 2024-04-28T12:40:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.072+0000 7f64f0ff9700 1 -- 172.21.15.53:0/4103939295 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f64e800b7e0 con 0x7f64f4107650 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.073+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f4064a80 con 0x7f64f4107650 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.075+0000 7f64f0ff9700 1 -- 172.21.15.53:0/4103939295 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f64e801a9e0 con 0x7f64f4107650 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.149+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f64f402e7d0 con 0x7f64f4107650 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.150+0000 7f64f0ff9700 1 -- 172.21.15.53:0/4103939295 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f64e801a430 con 0x7f64f4107650 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.151+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 msgr2=0x7f64f419eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.151+0000 7f64fa0ba700 1 --2- 172.21.15.53:0/4103939295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f419eea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f64e8011ac0 tx=0x7f64e8011ba0 comp rx=0 tx=0).stop 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.151+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 shutdown_connections 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.151+0000 7f64fa0ba700 1 --2- 172.21.15.53:0/4103939295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64f4107650 0x7f64f419eea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.151+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 >> 172.21.15.53:0/4103939295 conn(0x7f64f4103230 msgr2=0x7f64f4195b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.151+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 shutdown_connections 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:50.151+0000 7f64fa0ba700 1 -- 172.21.15.53:0/4103939295 wait complete. 2024-04-28T12:40:50.754 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (2/15)... 2024-04-28T12:40:51.083 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:50.774+0000 7fb6fd3c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:40:51.083 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:50.875+0000 7fb6fd3c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:40:51.083 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:51.086+0000 7fb6fd3c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:40:51.895 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:51.632+0000 7fb6fd3c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:40:51.895 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:51.729+0000 7fb6fd3c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:40:52.342 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:51.985+0000 7fb6fd3c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:40:52.342 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:52.124+0000 7fb6fd3c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:40:52.342 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:52.224+0000 7fb6fd3c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:40:52.342 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:52.345+0000 7fb6fd3c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:40:52.974 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:52.656+0000 7fb6fd3c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:40:53.529 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:53 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3067064123' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:40:53.529 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:53 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:53.300+0000 7fb6fd3c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:40:53.806 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:53 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:53.533+0000 7fb6fd3c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:40:53.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:40:53.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:53.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:53.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:53.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:40:39.138129+0000", 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.175+0000 7f980f7c7700 1 Processor -- start 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.175+0000 7f980f7c7700 1 -- start start 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.175+0000 7f980f7c7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f9808109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.175+0000 7f980f7c7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9808109760 con 0x7f9808108d70 2024-04-28T12:40:53.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.175+0000 7f980d563700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f9808109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.175+0000 7f980d563700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f9808109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52368/0 (socket says 172.21.15.53:52368) 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.175+0000 7f980d563700 1 -- 172.21.15.53:0/3547316202 learned_addr learned my addr 172.21.15.53:0/3547316202 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.176+0000 7f980d563700 1 -- 172.21.15.53:0/3547316202 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9808109f70 con 0x7f9808108d70 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.176+0000 7f980d563700 1 --2- 172.21.15.53:0/3547316202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f9808109190 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f98040098a0 tx=0x7f9804009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298bcda30ca48808 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.176+0000 7f97fffff700 1 -- 172.21.15.53:0/3547316202 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9804004160 con 0x7f9808108d70 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.176+0000 7f97fffff700 1 -- 172.21.15.53:0/3547316202 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9804004300 con 0x7f9808108d70 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.176+0000 7f97fffff700 1 -- 172.21.15.53:0/3547316202 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9804011af0 con 0x7f9808108d70 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 -- 172.21.15.53:0/3547316202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 msgr2=0x7f9808109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 --2- 172.21.15.53:0/3547316202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f9808109190 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f98040098a0 tx=0x7f9804009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 -- 172.21.15.53:0/3547316202 shutdown_connections 2024-04-28T12:40:53.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 --2- 172.21.15.53:0/3547316202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f9808109190 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 -- 172.21.15.53:0/3547316202 >> 172.21.15.53:0/3547316202 conn(0x7f98081042d0 msgr2=0x7f9808106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 -- 172.21.15.53:0/3547316202 shutdown_connections 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 -- 172.21.15.53:0/3547316202 wait complete. 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.177+0000 7f980f7c7700 1 Processor -- start 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980f7c7700 1 -- start start 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980f7c7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f980807eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980f7c7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f980810d320 con 0x7f9808108d70 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980d563700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f980807eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980d563700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f980807eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52380/0 (socket says 172.21.15.53:52380) 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980d563700 1 -- 172.21.15.53:0/3067064123 learned_addr learned my addr 172.21.15.53:0/3067064123 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980d563700 1 -- 172.21.15.53:0/3067064123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980807f0a0 con 0x7f9808108d70 2024-04-28T12:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f980d563700 1 --2- 172.21.15.53:0/3067064123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f980807eb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f98040117b0 tx=0x7f9804011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f97fe7fc700 1 -- 172.21.15.53:0/3067064123 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9804003af0 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.178+0000 7f97fe7fc700 1 -- 172.21.15.53:0/3067064123 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9804020070 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.179+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980807d290 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.179+0000 7f97fe7fc700 1 -- 172.21.15.53:0/3067064123 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9804019630 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.179+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f980807d7d0 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.179+0000 7f97fe7fc700 1 -- 172.21.15.53:0/3067064123 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9804019c80 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.179+0000 7f97fe7fc700 1 -- 172.21.15.53:0/3067064123 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f980400b720 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.179+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9808064a80 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.182+0000 7f97fe7fc700 1 -- 172.21.15.53:0/3067064123 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f98040042a0 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.257+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f980807da80 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.258+0000 7f97fe7fc700 1 -- 172.21.15.53:0/3067064123 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f980401a9c0 con 0x7f9808108d70 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.259+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 msgr2=0x7f980807eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:53.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.259+0000 7f980f7c7700 1 --2- 172.21.15.53:0/3067064123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f980807eb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f98040117b0 tx=0x7f9804011b70 comp rx=0 tx=0).stop 2024-04-28T12:40:53.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.259+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 shutdown_connections 2024-04-28T12:40:53.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.259+0000 7f980f7c7700 1 --2- 172.21.15.53:0/3067064123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9808108d70 0x7f980807eb60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:53.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.259+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 >> 172.21.15.53:0/3067064123 conn(0x7f98081042d0 msgr2=0x7f9808104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:53.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.259+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 shutdown_connections 2024-04-28T12:40:53.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:53.259+0000 7f980f7c7700 1 -- 172.21.15.53:0/3067064123 wait complete. 2024-04-28T12:40:53.829 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (3/15)... 2024-04-28T12:40:54.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: Activating manager daemon x 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: mgrmap e2: x(active, starting, since 0.00311934s) 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: Manager daemon x is now available 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' 2024-04-28T12:40:54.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:54 smithi053 ceph-mon[29182]: from='mgr.14100 172.21.15.53:0/3056231521' entity='mgr.x' 2024-04-28T12:40:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:55 smithi053 ceph-mon[29182]: mgrmap e3: x(active, since 1.00708s) 2024-04-28T12:40:56.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:56 smithi053 ceph-mon[29182]: mgrmap e4: x(active, since 2s) 2024-04-28T12:40:57.177 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:57.177 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:40:57.177 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:40:57.177 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-28T12:40:57.178 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:57.179 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:40:57.180 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:40:39.138129+0000", 2024-04-28T12:40:57.181 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.240+0000 7fb235833700 1 Processor -- start 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.240+0000 7fb235833700 1 -- start start 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.241+0000 7fb235833700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb2301091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.241+0000 7fb235833700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301097c0 con 0x7fb230108dd0 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.241+0000 7fb22effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb2301091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.241+0000 7fb22effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb2301091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52454/0 (socket says 172.21.15.53:52454) 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.241+0000 7fb22effd700 1 -- 172.21.15.53:0/802605103 learned_addr learned my addr 172.21.15.53:0/802605103 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.241+0000 7fb22effd700 1 -- 172.21.15.53:0/802605103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb230109f80 con 0x7fb230108dd0 2024-04-28T12:40:57.182 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.241+0000 7fb22effd700 1 --2- 172.21.15.53:0/802605103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb2301091f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2200098a0 tx=0x7fb220009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b546de21acd9a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb22dffb700 1 -- 172.21.15.53:0/802605103 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb220004160 con 0x7fb230108dd0 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb22dffb700 1 -- 172.21.15.53:0/802605103 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb220004300 con 0x7fb230108dd0 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb22dffb700 1 -- 172.21.15.53:0/802605103 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb220011af0 con 0x7fb230108dd0 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb235833700 1 -- 172.21.15.53:0/802605103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 msgr2=0x7fb2301091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb235833700 1 --2- 172.21.15.53:0/802605103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb2301091f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2200098a0 tx=0x7fb220009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb235833700 1 -- 172.21.15.53:0/802605103 shutdown_connections 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb235833700 1 --2- 172.21.15.53:0/802605103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb2301091f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb235833700 1 -- 172.21.15.53:0/802605103 >> 172.21.15.53:0/802605103 conn(0x7fb230104370 msgr2=0x7fb2301067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb235833700 1 -- 172.21.15.53:0/802605103 shutdown_connections 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.242+0000 7fb235833700 1 -- 172.21.15.53:0/802605103 wait complete. 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.243+0000 7fb235833700 1 Processor -- start 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.243+0000 7fb235833700 1 -- start start 2024-04-28T12:40:57.183 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.243+0000 7fb235833700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb23007f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.243+0000 7fb235833700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb23010d330 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.243+0000 7fb22effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb23007f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.243+0000 7fb22effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb23007f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52470/0 (socket says 172.21.15.53:52470) 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.243+0000 7fb22effd700 1 -- 172.21.15.53:0/2439615064 learned_addr learned my addr 172.21.15.53:0/2439615064 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.244+0000 7fb22effd700 1 -- 172.21.15.53:0/2439615064 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23007f930 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.244+0000 7fb22effd700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb23007f3f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb220009fb0 tx=0x7fb220011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.244+0000 7fb234831700 1 -- 172.21.15.53:0/2439615064 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2200036a0 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.244+0000 7fb234831700 1 -- 172.21.15.53:0/2439615064 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb22001a430 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.244+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23007db20 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.244+0000 7fb234831700 1 -- 172.21.15.53:0/2439615064 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb220019550 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.244+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23007e060 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.245+0000 7fb234831700 1 -- 172.21.15.53:0/2439615064 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fb220019850 con 0x7fb230108dd0 2024-04-28T12:40:57.184 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.245+0000 7fb234831700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7fb21c039510 0x7fb21c03b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.245+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb230064a80 con 0x7fb230108dd0 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.246+0000 7fb234831700 1 -- 172.21.15.53:0/2439615064 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb22004d890 con 0x7fb230108dd0 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.246+0000 7fb22e7fc700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7fb21c039510 0x7fb21c03b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.246+0000 7fb22e7fc700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7fb21c039510 0x7fb21c03b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb218006fd0 tx=0x7fb218006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.251+0000 7fb234831700 1 -- 172.21.15.53:0/2439615064 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb220020240 con 0x7fb230108dd0 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.546+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb23007e970 con 0x7fb230108dd0 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.546+0000 7fb234831700 1 -- 172.21.15.53:0/2439615064 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fb22004d2b0 con 0x7fb230108dd0 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7fb21c039510 msgr2=0x7fb21c03b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7fb21c039510 0x7fb21c03b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb218006fd0 tx=0x7fb218006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 msgr2=0x7fb23007f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb23007f3f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb220009fb0 tx=0x7fb220011940 comp rx=0 tx=0).stop 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 shutdown_connections 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7fb21c039510 0x7fb21c03b9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 --2- 172.21.15.53:0/2439615064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb230108dd0 0x7fb23007f3f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:57.185 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 >> 172.21.15.53:0/2439615064 conn(0x7fb230104370 msgr2=0x7fb230104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:57.186 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 shutdown_connections 2024-04-28T12:40:57.186 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:56.549+0000 7fb235833700 1 -- 172.21.15.53:0/2439615064 wait complete. 2024-04-28T12:40:57.186 INFO:teuthology.orchestra.run.smithi053.stdout:mgr is available 2024-04-28T12:40:57.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:57 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2439615064' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:40:58.420 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:58.420 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [global] 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout fsid = 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [osd] 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-28T12:40:58.421 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.603+0000 7f51febf8700 1 Processor -- start 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.603+0000 7f51febf8700 1 -- start start 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.603+0000 7f51febf8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f81080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.603+0000 7f51febf8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f51f8108680 con 0x7f51f8107c90 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51fc994700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f81080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51fc994700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f81080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46982/0 (socket says 172.21.15.53:46982) 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51fc994700 1 -- 172.21.15.53:0/502337898 learned_addr learned my addr 172.21.15.53:0/502337898 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51fc994700 1 -- 172.21.15.53:0/502337898 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f8108e90 con 0x7f51f8107c90 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51fc994700 1 --2- 172.21.15.53:0/502337898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f81080b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f51e80098a0 tx=0x7f51e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e00f2f3e89ae006 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:58.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51f77fe700 1 -- 172.21.15.53:0/502337898 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e8004160 con 0x7f51f8107c90 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51f77fe700 1 -- 172.21.15.53:0/502337898 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f51e8004300 con 0x7f51f8107c90 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.604+0000 7f51f77fe700 1 -- 172.21.15.53:0/502337898 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e8011af0 con 0x7f51f8107c90 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 -- 172.21.15.53:0/502337898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 msgr2=0x7f51f81080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 --2- 172.21.15.53:0/502337898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f81080b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f51e80098a0 tx=0x7f51e8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 -- 172.21.15.53:0/502337898 shutdown_connections 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 --2- 172.21.15.53:0/502337898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f81080b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 -- 172.21.15.53:0/502337898 >> 172.21.15.53:0/502337898 conn(0x7f51f81031f0 msgr2=0x7f51f8105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 -- 172.21.15.53:0/502337898 shutdown_connections 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 -- 172.21.15.53:0/502337898 wait complete. 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.605+0000 7f51febf8700 1 Processor -- start 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51febf8700 1 -- start start 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51febf8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51febf8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f51f810c240 con 0x7f51f8107c90 2024-04-28T12:40:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51fc994700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51fc994700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46996/0 (socket says 172.21.15.53:46996) 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51fc994700 1 -- 172.21.15.53:0/1579332078 learned_addr learned my addr 172.21.15.53:0/1579332078 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51fc994700 1 -- 172.21.15.53:0/1579332078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f819c180 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51fc994700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f819bc40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009fb0 tx=0x7f51e8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51f5ffb700 1 -- 172.21.15.53:0/1579332078 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e80036a0 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51f5ffb700 1 -- 172.21.15.53:0/1579332078 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f51e801a430 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f819c410 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.606+0000 7f51f5ffb700 1 -- 172.21.15.53:0/1579332078 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e8019550 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.607+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f819f100 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.607+0000 7f51f5ffb700 1 -- 172.21.15.53:0/1579332078 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f51e8019850 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.608+0000 7f51f5ffb700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f51e4039510 0x7f51e403b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.608+0000 7f51f5ffb700 1 -- 172.21.15.53:0/1579332078 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f51e804dbb0 con 0x7f51f8107c90 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.608+0000 7f51f7fff700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f51e4039510 0x7f51e403b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:58.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.608+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f8064a80 con 0x7f51f8107c90 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.608+0000 7f51f7fff700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f51e4039510 0x7f51e403b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51e000ad30 tx=0x7f51e00093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.613+0000 7f51f5ffb700 1 -- 172.21.15.53:0/1579332078 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f51e8023c90 con 0x7f51f8107c90 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.821+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f51f819f2d0 con 0x7f51f8107c90 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.822+0000 7f51f5ffb700 1 -- 172.21.15.53:0/1579332078 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f51e8032020 con 0x7f51f8107c90 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f51e4039510 msgr2=0x7f51e403b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f51e4039510 0x7f51e403b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f51e000ad30 tx=0x7f51e00093f0 comp rx=0 tx=0).stop 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 msgr2=0x7f51f819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f819bc40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009fb0 tx=0x7f51e8011940 comp rx=0 tx=0).stop 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 shutdown_connections 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f51e4039510 0x7f51e403b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 --2- 172.21.15.53:0/1579332078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f8107c90 0x7f51f819bc40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 >> 172.21.15.53:0/1579332078 conn(0x7f51f81031f0 msgr2=0x7f51f8103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 shutdown_connections 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:57.824+0000 7f51febf8700 1 -- 172.21.15.53:0/1579332078 wait complete. 2024-04-28T12:40:58.425 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling cephadm module... 2024-04-28T12:40:58.552 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:58 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1579332078' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-28T12:40:59.865 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:40:59 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1327238071' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-28T12:40:59.866 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: ignoring --setuser ceph since I am not root 2024-04-28T12:40:59.866 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: ignoring --setgroup ceph since I am not root 2024-04-28T12:40:59.866 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:59.724+0000 7f1a24b61200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:41:00.098 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.886+0000 7f5ffa99f700 1 Processor -- start 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.886+0000 7f5ffa99f700 1 -- start start 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ffa99f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff410bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ffa99f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff407cea0 con 0x7f5ff4109800 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff410bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff410bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47000/0 (socket says 172.21.15.53:47000) 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ff3fff700 1 -- 172.21.15.53:0/3905147853 learned_addr learned my addr 172.21.15.53:0/3905147853 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ff3fff700 1 -- 172.21.15.53:0/3905147853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff410c130 con 0x7f5ff4109800 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ff3fff700 1 --2- 172.21.15.53:0/3905147853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff410bbf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40098a0 tx=0x7f5fe4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2159359eca3a40d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.887+0000 7f5ff2ffd700 1 -- 172.21.15.53:0/3905147853 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fe4004160 con 0x7f5ff4109800 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ff2ffd700 1 -- 172.21.15.53:0/3905147853 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5fe4004300 con 0x7f5ff4109800 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ff2ffd700 1 -- 172.21.15.53:0/3905147853 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fe4011af0 con 0x7f5ff4109800 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ffa99f700 1 -- 172.21.15.53:0/3905147853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 msgr2=0x7f5ff410bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:00.099 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ffa99f700 1 --2- 172.21.15.53:0/3905147853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff410bbf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40098a0 tx=0x7f5fe4009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ffa99f700 1 -- 172.21.15.53:0/3905147853 shutdown_connections 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ffa99f700 1 --2- 172.21.15.53:0/3905147853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff410bbf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ffa99f700 1 -- 172.21.15.53:0/3905147853 >> 172.21.15.53:0/3905147853 conn(0x7f5ff41031f0 msgr2=0x7f5ff4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ffa99f700 1 -- 172.21.15.53:0/3905147853 shutdown_connections 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.888+0000 7f5ffa99f700 1 -- 172.21.15.53:0/3905147853 wait complete. 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.889+0000 7f5ffa99f700 1 Processor -- start 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.889+0000 7f5ffa99f700 1 -- start start 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.889+0000 7f5ffa99f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.889+0000 7f5ffa99f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff410d100 con 0x7f5ff4109800 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.889+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.889+0000 7f5ff3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47014/0 (socket says 172.21.15.53:47014) 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.889+0000 7f5ff3fff700 1 -- 172.21.15.53:0/1327238071 learned_addr learned my addr 172.21.15.53:0/1327238071 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.890+0000 7f5ff3fff700 1 -- 172.21.15.53:0/1327238071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff419d220 con 0x7f5ff4109800 2024-04-28T12:41:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.890+0000 7f5ff3fff700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff419cce0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4011ac0 tx=0x7f5fe4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.890+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fe4003970 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.890+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff419d4b0 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.890+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5fe4003b10 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.890+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fe4024610 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.890+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff41a0190 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.891+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f5fe4024840 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.891+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff4064a80 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.891+0000 7f5ff17fa700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5fe00395b0 0x7f5fe003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.891+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5fe404da00 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.891+0000 7f5ff37fe700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5fe00395b0 0x7f5fe003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.892+0000 7f5ff37fe700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5fe00395b0 0x7f5fe003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc006fd0 tx=0x7f5fdc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:58.896+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5fe400ba20 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.169+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f5ff41a0360 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.545+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f5ff41a0360 con 0x7f5ff4109800 2024-04-28T12:41:00.101 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.547+0000 7f5ff17fa700 1 -- 172.21.15.53:0/1327238071 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f5fe402d770 con 0x7f5ff4109800 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5fe00395b0 msgr2=0x7f5fe003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5fe00395b0 0x7f5fe003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc006fd0 tx=0x7f5fdc006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 msgr2=0x7f5ff419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff419cce0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4011ac0 tx=0x7f5fe4011ba0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 shutdown_connections 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5fe00395b0 0x7f5fe003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 --2- 172.21.15.53:0/1327238071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff4109800 0x7f5ff419cce0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 >> 172.21.15.53:0/1327238071 conn(0x7f5ff41031f0 msgr2=0x7f5ff4103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 shutdown_connections 2024-04-28T12:41:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:59.549+0000 7f5ffa99f700 1 -- 172.21.15.53:0/1327238071 wait complete. 2024-04-28T12:41:00.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:40:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:40:59.916+0000 7f1a24b61200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:00.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:00.023+0000 7f1a24b61200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:41:00.440 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:00.355+0000 7f1a24b61200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:41:00.440 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:00.444+0000 7f1a24b61200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:41:00.769 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:00 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1327238071' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-28T12:41:00.769 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:00 smithi053 ceph-mon[29182]: mgrmap e5: x(active, since 6s) 2024-04-28T12:41:01.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:41:01.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-28T12:41:01.422 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc788d700 1 Processor -- start 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc788d700 1 -- start start 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc788d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc010a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc788d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc010ad80 con 0x7f5cc010a390 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc5629700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc010a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc5629700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc010a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47046/0 (socket says 172.21.15.53:47046) 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc5629700 1 -- 172.21.15.53:0/2809941484 learned_addr learned my addr 172.21.15.53:0/2809941484 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.567+0000 7f5cc5629700 1 -- 172.21.15.53:0/2809941484 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc010af00 con 0x7f5cc010a390 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cc5629700 1 --2- 172.21.15.53:0/2809941484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc010a7b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0098a0 tx=0x7f5cbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838a6b7edb072089 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cb7fff700 1 -- 172.21.15.53:0/2809941484 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cbc004160 con 0x7f5cc010a390 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cb7fff700 1 -- 172.21.15.53:0/2809941484 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5cbc004300 con 0x7f5cc010a390 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cb7fff700 1 -- 172.21.15.53:0/2809941484 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cbc011af0 con 0x7f5cc010a390 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cc788d700 1 -- 172.21.15.53:0/2809941484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 msgr2=0x7f5cc010a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cc788d700 1 --2- 172.21.15.53:0/2809941484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc010a7b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0098a0 tx=0x7f5cbc009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cc788d700 1 -- 172.21.15.53:0/2809941484 shutdown_connections 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cc788d700 1 --2- 172.21.15.53:0/2809941484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc010a7b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.568+0000 7f5cc788d700 1 -- 172.21.15.53:0/2809941484 >> 172.21.15.53:0/2809941484 conn(0x7f5cc0102970 msgr2=0x7f5cc0104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.569+0000 7f5cc788d700 1 -- 172.21.15.53:0/2809941484 shutdown_connections 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.569+0000 7f5cc788d700 1 -- 172.21.15.53:0/2809941484 wait complete. 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.569+0000 7f5cc788d700 1 Processor -- start 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.569+0000 7f5cc788d700 1 -- start start 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc788d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc788d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc0107b90 con 0x7f5cc010a390 2024-04-28T12:41:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc5629700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc5629700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47056/0 (socket says 172.21.15.53:47056) 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc5629700 1 -- 172.21.15.53:0/167015289 learned_addr learned my addr 172.21.15.53:0/167015289 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc5629700 1 -- 172.21.15.53:0/167015289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc019d210 con 0x7f5cc010a390 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc5629700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc019ccd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009fb0 tx=0x7f5cbc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cb67fc700 1 -- 172.21.15.53:0/167015289 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cbc0036a0 con 0x7f5cc010a390 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cb67fc700 1 -- 172.21.15.53:0/167015289 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5cbc01a430 con 0x7f5cc010a390 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc019d500 con 0x7f5cc010a390 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.570+0000 7f5cb67fc700 1 -- 172.21.15.53:0/167015289 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cbc019550 con 0x7f5cc010a390 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.571+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc01a0180 con 0x7f5cc010a390 2024-04-28T12:41:01.425 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.571+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cc0064a80 con 0x7f5cc010a390 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.572+0000 7f5cb67fc700 1 -- 172.21.15.53:0/167015289 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f5cbc019850 con 0x7f5cc010a390 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.572+0000 7f5cb67fc700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 0x7f5cac03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.572+0000 7f5cb67fc700 1 -- 172.21.15.53:0/167015289 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5cbc04da20 con 0x7f5cc010a390 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.572+0000 7f5cc4e28700 1 -- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 msgr2=0x7f5cac03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3936438773 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.572+0000 7f5cc4e28700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 0x7f5cac03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.577+0000 7f5cb67fc700 1 -- 172.21.15.53:0/167015289 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5cbc04d440 con 0x7f5cc010a390 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.773+0000 7f5cc4e28700 1 -- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 msgr2=0x7f5cac03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3936438773 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.773+0000 7f5cc4e28700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 0x7f5cac03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.870+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f5cc01086b0 con 0x7f5cc010a390 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.871+0000 7f5cb67fc700 1 -- 172.21.15.53:0/167015289 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f5cbc01a5e0 con 0x7f5cc010a390 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 msgr2=0x7f5cac03ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:41:01.426 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 0x7f5cac03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 msgr2=0x7f5cc019ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc019ccd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009fb0 tx=0x7f5cbc011940 comp rx=0 tx=0).stop 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 shutdown_connections 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f5cac039560 0x7f5cac03ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 --2- 172.21.15.53:0/167015289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cc010a390 0x7f5cc019ccd0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009fb0 tx=0x7f5cbc011940 comp rx=0 tx=0).stop 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.875+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 >> 172.21.15.53:0/167015289 conn(0x7f5cc0102970 msgr2=0x7f5cc01035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.884+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 shutdown_connections 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:00.884+0000 7f5cc788d700 1 -- 172.21.15.53:0/167015289 wait complete. 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for the mgr to restart... 2024-04-28T12:41:01.427 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr epoch 5... 2024-04-28T12:41:01.800 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:01 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/167015289' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-28T12:41:01.800 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:01 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:01.712+0000 7f1a24b61200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:41:02.080 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:01 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:01.895+0000 7f1a24b61200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:02.080 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:02.083+0000 7f1a24b61200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:41:02.387 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:02.390+0000 7f1a24b61200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:41:02.645 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:02.488+0000 7f1a24b61200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:41:02.949 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:02.699+0000 7f1a24b61200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:41:02.949 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:02.900+0000 7f1a24b61200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:03.645 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:03 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:03.526+0000 7f1a24b61200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:41:04.265 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:03 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:03.961+0000 7f1a24b61200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:41:04.266 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:04.061+0000 7f1a24b61200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:41:04.266 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:04.269+0000 7f1a24b61200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:41:05.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:04.812+0000 7f1a24b61200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:41:05.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:04.907+0000 7f1a24b61200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:41:05.510 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:05.159+0000 7f1a24b61200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:41:05.510 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:05.296+0000 7f1a24b61200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:41:05.510 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:05.394+0000 7f1a24b61200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:41:05.511 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:05.513+0000 7f1a24b61200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:06.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:05.825+0000 7f1a24b61200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:41:06.722 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:06.450+0000 7f1a24b61200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:41:06.722 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:06.672+0000 7f1a24b61200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:41:07.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: Active manager daemon x restarted 2024-04-28T12:41:07.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: Activating manager daemon x 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: osdmap e2: 0 total, 0 up, 0 in 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: mgrmap e6: x(active, starting, since 0.00344764s) 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: Manager daemon x is now available 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:07.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:06 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T12:41:07.995 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:07 smithi053 ceph-mon[29182]: Found migration_current of "None". Setting to last migration. 2024-04-28T12:41:07.996 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:07 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T12:41:07.996 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:07 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:07.996 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:07 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:07.996 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:07 smithi053 ceph-mon[29182]: mgrmap e7: x(active, since 1.00562s) 2024-04-28T12:41:08.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.909+0000 7f53718cd700 1 Processor -- start 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.909+0000 7f53718cd700 1 -- start start 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.909+0000 7f53718cd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c07d570 0x7f536c10b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.909+0000 7f53718cd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f536c10bec0 con 0x7f536c07d570 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.909+0000 7f536affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c07d570 0x7f536c10b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.909+0000 7f536affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c07d570 0x7f536c10b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47060/0 (socket says 172.21.15.53:47060) 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.909+0000 7f536affd700 1 -- 172.21.15.53:0/1999961720 learned_addr learned my addr 172.21.15.53:0/1999961720 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:08.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.910+0000 7f536affd700 1 -- 172.21.15.53:0/1999961720 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536c10c040 con 0x7f536c07d570 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.910+0000 7f536affd700 1 --2- 172.21.15.53:0/1999961720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c07d570 0x7f536c10b8f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f535c0098a0 tx=0x7f535c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e2b5a6ac7e655b server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.910+0000 7f5369ffb700 1 -- 172.21.15.53:0/1999961720 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f535c004160 con 0x7f536c07d570 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.910+0000 7f5369ffb700 1 -- 172.21.15.53:0/1999961720 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f535c004300 con 0x7f536c07d570 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.910+0000 7f5369ffb700 1 -- 172.21.15.53:0/1999961720 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f535c011af0 con 0x7f536c07d570 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.911+0000 7f53718cd700 1 -- 172.21.15.53:0/1999961720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c07d570 msgr2=0x7f536c10b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.911+0000 7f53718cd700 1 --2- 172.21.15.53:0/1999961720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c07d570 0x7f536c10b8f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f535c0098a0 tx=0x7f535c009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.911+0000 7f53718cd700 1 -- 172.21.15.53:0/1999961720 shutdown_connections 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.911+0000 7f53718cd700 1 --2- 172.21.15.53:0/1999961720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c07d570 0x7f536c10b8f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.911+0000 7f53718cd700 1 -- 172.21.15.53:0/1999961720 >> 172.21.15.53:0/1999961720 conn(0x7f536c103230 msgr2=0x7f536c105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.911+0000 7f53718cd700 1 -- 172.21.15.53:0/1999961720 shutdown_connections 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.912+0000 7f53718cd700 1 -- 172.21.15.53:0/1999961720 wait complete. 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.912+0000 7f53718cd700 1 Processor -- start 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.913+0000 7f53718cd700 1 -- start start 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.913+0000 7f53718cd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c19cc60 0x7f536c19d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.913+0000 7f53718cd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f536c10d0c0 con 0x7f536c19cc60 2024-04-28T12:41:08.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.913+0000 7f536affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c19cc60 0x7f536c19d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.913+0000 7f536affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c19cc60 0x7f536c19d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47072/0 (socket says 172.21.15.53:47072) 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.913+0000 7f536affd700 1 -- 172.21.15.53:0/1088074745 learned_addr learned my addr 172.21.15.53:0/1088074745 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.913+0000 7f536affd700 1 -- 172.21.15.53:0/1088074745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536c19d5c0 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.914+0000 7f536affd700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c19cc60 0x7f536c19d080 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f535c009fb0 tx=0x7f535c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.914+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f535c0036a0 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.914+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f535c01a430 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.914+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536c19d850 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.914+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f535c019550 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.914+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536c1a0330 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.915+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f535c019800 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.915+0000 7f53708cb700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.915+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 --> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f535403c180 con 0x7f53540395b0 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.915+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f535c04d8e0 con 0x7f536c19cc60 2024-04-28T12:41:08.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.915+0000 7f536a7fc700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 msgr2=0x7f535403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3936438773 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:01.915+0000 7f536a7fc700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:02.116+0000 7f536a7fc700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 msgr2=0x7f535403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3936438773 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:02.116+0000 7f536a7fc700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:02.517+0000 7f536a7fc700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 msgr2=0x7f535403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3936438773 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:02.517+0000 7f536a7fc700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:03.318+0000 7f536a7fc700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 msgr2=0x7f535403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3936438773 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:03.318+0000 7f536a7fc700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:04.919+0000 7f536a7fc700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 msgr2=0x7f535403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3936438773 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:04.919+0000 7f536a7fc700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:06.676+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f535c019b40 con 0x7f536c19cc60 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:06.676+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 msgr2=0x7f535403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:06.676+0000 7f53708cb700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.677+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f535c04f710 con 0x7f536c19cc60 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.677+0000 7f53708cb700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f535403cea0 0x7f535403f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:08.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.677+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f535c001d30 con 0x7f535403cea0 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.678+0000 7f536a7fc700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f535403cea0 0x7f535403f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.678+0000 7f536a7fc700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f535403cea0 0x7f535403f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5360003a10 tx=0x7f53600092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.679+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f535c001d30 con 0x7f535403cea0 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.687+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f536c06a540 con 0x7f535403cea0 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.687+0000 7f53708cb700 1 -- 172.21.15.53:0/1088074745 <== mgr.14120 v2:172.21.15.53:6800/2275002354 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f536c06a540 con 0x7f535403cea0 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f535403cea0 msgr2=0x7f535403f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f535403cea0 0x7f535403f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5360003a10 tx=0x7f53600092b0 comp rx=0 tx=0).stop 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c19cc60 msgr2=0x7f536c19d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c19cc60 0x7f536c19d080 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f535c009fb0 tx=0x7f535c011940 comp rx=0 tx=0).stop 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 shutdown_connections 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f535403cea0 0x7f535403f290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536c19cc60 0x7f536c19d080 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 --2- 172.21.15.53:0/1088074745 >> [v2:172.21.15.53:6800/3936438773,v1:172.21.15.53:6801/3936438773] conn(0x7f53540395b0 0x7f535403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:08.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 >> 172.21.15.53:0/1088074745 conn(0x7f536c103230 msgr2=0x7f536c103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:08.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 shutdown_connections 2024-04-28T12:41:08.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:07.688+0000 7f53718cd700 1 -- 172.21.15.53:0/1088074745 wait complete. 2024-04-28T12:41:08.295 INFO:teuthology.orchestra.run.smithi053.stdout:mgr epoch 5 is available 2024-04-28T12:41:08.295 INFO:teuthology.orchestra.run.smithi053.stdout:Setting orchestrator backend to cephadm... 2024-04-28T12:41:08.892 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:08 smithi053 ceph-mon[29182]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-28T12:41:08.892 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:08 smithi053 ceph-mon[29182]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-28T12:41:08.892 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:08 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:07] ENGINE Bus STARTING 2024-04-28T12:41:08.892 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:08 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:07] ENGINE Serving on http://172.21.15.53:8765 2024-04-28T12:41:08.892 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:08 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:08] ENGINE Serving on https://172.21.15.53:7150 2024-04-28T12:41:08.892 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:08 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:08] ENGINE Bus STARTED 2024-04-28T12:41:08.892 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:08 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.658+0000 7f2023a63700 1 Processor -- start 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.658+0000 7f2023a63700 1 -- start start 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f2023a63700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c109170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f2023a63700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f201c109740 con 0x7f201c108d50 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f20217ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f20217ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51484/0 (socket says 172.21.15.53:51484) 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f20217ff700 1 -- 172.21.15.53:0/794521508 learned_addr learned my addr 172.21.15.53:0/794521508 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:09.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f20217ff700 1 -- 172.21.15.53:0/794521508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c109f50 con 0x7f201c108d50 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f20217ff700 1 --2- 172.21.15.53:0/794521508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c109170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f20180098a0 tx=0x7f2018009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ce68658d18d51b server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.659+0000 7f2013fff700 1 -- 172.21.15.53:0/794521508 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2018004160 con 0x7f201c108d50 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2013fff700 1 -- 172.21.15.53:0/794521508 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2018004300 con 0x7f201c108d50 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2013fff700 1 -- 172.21.15.53:0/794521508 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2018011af0 con 0x7f201c108d50 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2023a63700 1 -- 172.21.15.53:0/794521508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 msgr2=0x7f201c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2023a63700 1 --2- 172.21.15.53:0/794521508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c109170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f20180098a0 tx=0x7f2018009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2023a63700 1 -- 172.21.15.53:0/794521508 shutdown_connections 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2023a63700 1 --2- 172.21.15.53:0/794521508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c109170 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2023a63700 1 -- 172.21.15.53:0/794521508 >> 172.21.15.53:0/794521508 conn(0x7f201c1042d0 msgr2=0x7f201c106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2023a63700 1 -- 172.21.15.53:0/794521508 shutdown_connections 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.660+0000 7f2023a63700 1 -- 172.21.15.53:0/794521508 wait complete. 2024-04-28T12:41:09.402 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.661+0000 7f2023a63700 1 Processor -- start 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.661+0000 7f2023a63700 1 -- start start 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.661+0000 7f2023a63700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.661+0000 7f2023a63700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f201c10d300 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.661+0000 7f20217ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.661+0000 7f20217ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c19cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51486/0 (socket says 172.21.15.53:51486) 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.661+0000 7f20217ff700 1 -- 172.21.15.53:0/890276081 learned_addr learned my addr 172.21.15.53:0/890276081 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.662+0000 7f20217ff700 1 -- 172.21.15.53:0/890276081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c19d240 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.662+0000 7f20217ff700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c19cd00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2018009fb0 tx=0x7f2018011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.662+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20180036a0 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.662+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f201801a430 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.662+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c19d4d0 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.662+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2018019550 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.662+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c1a01b0 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.663+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f2018019800 con 0x7f201c108d50 2024-04-28T12:41:09.403 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.663+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c064a80 con 0x7f201c108d50 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.663+0000 7f20127fc700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f2008039480 0x7f200803b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.663+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f201804d710 con 0x7f201c108d50 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.663+0000 7f2020ffe700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f2008039480 0x7f200803b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.664+0000 7f2020ffe700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f2008039480 0x7f200803b940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f200c006fd0 tx=0x7f200c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.668+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f201800fda0 con 0x7f201c108d50 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.730+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f201802d440 con 0x7f201c108d50 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.895+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f201c1055b0 con 0x7f2008039480 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.900+0000 7f20127fc700 1 -- 172.21.15.53:0/890276081 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f201c1055b0 con 0x7f2008039480 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f2008039480 msgr2=0x7f200803b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f2008039480 0x7f200803b940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f200c006fd0 tx=0x7f200c006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 msgr2=0x7f201c19cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c19cd00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2018009fb0 tx=0x7f2018011940 comp rx=0 tx=0).stop 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 shutdown_connections 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f2008039480 0x7f200803b940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 --2- 172.21.15.53:0/890276081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f201c108d50 0x7f201c19cd00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:09.404 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 >> 172.21.15.53:0/890276081 conn(0x7f201c1042d0 msgr2=0x7f201c104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:09.405 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 shutdown_connections 2024-04-28T12:41:09.405 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:08.903+0000 7f2023a63700 1 -- 172.21.15.53:0/890276081 wait complete. 2024-04-28T12:41:09.820 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:09 smithi053 ceph-mon[29182]: mgrmap e8: x(active, since 2s) 2024-04-28T12:41:09.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:09 smithi053 ceph-mon[29182]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:09.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:09 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:09.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:09 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.849+0000 7fee1ff89700 1 Processor -- start 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.849+0000 7fee1ff89700 1 -- start start 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.849+0000 7fee1ff89700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1807db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.849+0000 7fee1ff89700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee1807e150 con 0x7fee1807f6d0 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51502/0 (socket says 172.21.15.53:51502) 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1dd25700 1 -- 172.21.15.53:0/1811274589 learned_addr learned my addr 172.21.15.53:0/1811274589 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:10.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1dd25700 1 -- 172.21.15.53:0/1811274589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1807e2d0 con 0x7fee1807f6d0 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1dd25700 1 --2- 172.21.15.53:0/1811274589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1807db80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee08009e80 tx=0x7fee0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b43bb6caaa4e07 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1cd23700 1 -- 172.21.15.53:0/1811274589 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee08017070 con 0x7fee1807f6d0 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1cd23700 1 -- 172.21.15.53:0/1811274589 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fee0800b8c0 con 0x7fee1807f6d0 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.850+0000 7fee1cd23700 1 -- 172.21.15.53:0/1811274589 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee08004490 con 0x7fee1807f6d0 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.851+0000 7fee1ff89700 1 -- 172.21.15.53:0/1811274589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 msgr2=0x7fee1807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.851+0000 7fee1ff89700 1 --2- 172.21.15.53:0/1811274589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1807db80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee08009e80 tx=0x7fee0800b330 comp rx=0 tx=0).stop 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.851+0000 7fee1ff89700 1 -- 172.21.15.53:0/1811274589 shutdown_connections 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.851+0000 7fee1ff89700 1 --2- 172.21.15.53:0/1811274589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1807db80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.851+0000 7fee1ff89700 1 -- 172.21.15.53:0/1811274589 >> 172.21.15.53:0/1811274589 conn(0x7fee18104370 msgr2=0x7fee181067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.851+0000 7fee1ff89700 1 -- 172.21.15.53:0/1811274589 shutdown_connections 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.851+0000 7fee1ff89700 1 -- 172.21.15.53:0/1811274589 wait complete. 2024-04-28T12:41:10.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1ff89700 1 Processor -- start 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1ff89700 1 -- start start 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1ff89700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1ff89700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee1810f320 con 0x7fee1807f6d0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51508/0 (socket says 172.21.15.53:51508) 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1dd25700 1 -- 172.21.15.53:0/3922479557 learned_addr learned my addr 172.21.15.53:0/3922479557 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1dd25700 1 -- 172.21.15.53:0/3922479557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1819f3c0 con 0x7fee1807f6d0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.852+0000 7fee1dd25700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1819ee80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee0800b7b0 tx=0x7fee0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.853+0000 7fee16ffd700 1 -- 172.21.15.53:0/3922479557 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee08020020 con 0x7fee1807f6d0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.853+0000 7fee16ffd700 1 -- 172.21.15.53:0/3922479557 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fee08012430 con 0x7fee1807f6d0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.853+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee1819f650 con 0x7fee1807f6d0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.853+0000 7fee16ffd700 1 -- 172.21.15.53:0/3922479557 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee08017040 con 0x7fee1807f6d0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.853+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee181a2340 con 0x7fee1807f6d0 2024-04-28T12:41:10.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.854+0000 7fee16ffd700 1 -- 172.21.15.53:0/3922479557 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fee08023020 con 0x7fee1807f6d0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.854+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee1806a540 con 0x7fee1807f6d0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.854+0000 7fee16ffd700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fee000395b0 0x7fee0003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.854+0000 7fee16ffd700 1 -- 172.21.15.53:0/3922479557 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fee0800f050 con 0x7fee1807f6d0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.854+0000 7fee1d524700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fee000395b0 0x7fee0003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.854+0000 7fee1d524700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fee000395b0 0x7fee0003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee0c006fd0 tx=0x7fee0c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:09.859+0000 7fee16ffd700 1 -- 172.21.15.53:0/3922479557 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fee0802a020 con 0x7fee1807f6d0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.067+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fee18105570 con 0x7fee000395b0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.068+0000 7fee16ffd700 1 -- 172.21.15.53:0/3922479557 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fee18105570 con 0x7fee000395b0 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fee000395b0 msgr2=0x7fee0003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fee000395b0 0x7fee0003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee0c006fd0 tx=0x7fee0c006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 msgr2=0x7fee1819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:10.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1819ee80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee0800b7b0 tx=0x7fee0800be20 comp rx=0 tx=0).stop 2024-04-28T12:41:10.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 shutdown_connections 2024-04-28T12:41:10.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fee000395b0 0x7fee0003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:10.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 --2- 172.21.15.53:0/3922479557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee1807f6d0 0x7fee1819ee80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:10.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 >> 172.21.15.53:0/3922479557 conn(0x7fee18104370 msgr2=0x7fee18104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:10.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 shutdown_connections 2024-04-28T12:41:10.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:10.070+0000 7fee1ff89700 1 -- 172.21.15.53:0/3922479557 wait complete. 2024-04-28T12:41:10.620 INFO:teuthology.orchestra.run.smithi053.stdout:Generating ssh key... 2024-04-28T12:41:11.010 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:10 smithi053 ceph-mon[29182]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:12.054 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-mon[29182]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-mon[29182]: Generating ssh key... 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: Generating public/private rsa key pair. 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: Your identification has been saved in /tmp/tmpyc4zuwqv/key. 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: Your public key has been saved in /tmp/tmpyc4zuwqv/key.pub. 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: The key fingerprint is: 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: SHA256:N1bIY/tbCGvx3Tfp/FO1/K5Z4mnisHW/RVN+3aVRoQc ceph-10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: The key's randomart image is: 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: +---[RSA 3072]----+ 2024-04-28T12:41:12.055 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | E .o| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | . . o. | 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | = ....o| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | . + .+*| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | S B o.O| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | o B o ==| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | + = =oB| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | . +.=+B=| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: | ..oo=*B| 2024-04-28T12:41:12.056 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: +----[SHA256]-----+ 2024-04-28T12:41:12.355 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.035+0000 7f88eb220700 1 Processor -- start 2024-04-28T12:41:12.355 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.035+0000 7f88eb220700 1 -- start start 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.035+0000 7f88eb220700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e41091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.035+0000 7f88eb220700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88e41097c0 con 0x7f88e4108dd0 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.035+0000 7f88e8fbc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e41091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.035+0000 7f88e8fbc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e41091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51514/0 (socket says 172.21.15.53:51514) 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.035+0000 7f88e8fbc700 1 -- 172.21.15.53:0/812791219 learned_addr learned my addr 172.21.15.53:0/812791219 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.036+0000 7f88e8fbc700 1 -- 172.21.15.53:0/812791219 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e4109f80 con 0x7f88e4108dd0 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.036+0000 7f88e8fbc700 1 --2- 172.21.15.53:0/812791219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e41091f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f88d80098a0 tx=0x7f88d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49793cb1c71bb1c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.036+0000 7f88e37fe700 1 -- 172.21.15.53:0/812791219 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88d8004160 con 0x7f88e4108dd0 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.036+0000 7f88e37fe700 1 -- 172.21.15.53:0/812791219 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88d8004300 con 0x7f88e4108dd0 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.036+0000 7f88e37fe700 1 -- 172.21.15.53:0/812791219 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88d8011af0 con 0x7f88e4108dd0 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.036+0000 7f88eb220700 1 -- 172.21.15.53:0/812791219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 msgr2=0x7f88e41091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.036+0000 7f88eb220700 1 --2- 172.21.15.53:0/812791219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e41091f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f88d80098a0 tx=0x7f88d8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 -- 172.21.15.53:0/812791219 shutdown_connections 2024-04-28T12:41:12.356 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 --2- 172.21.15.53:0/812791219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e41091f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 -- 172.21.15.53:0/812791219 >> 172.21.15.53:0/812791219 conn(0x7f88e4104370 msgr2=0x7f88e41067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 -- 172.21.15.53:0/812791219 shutdown_connections 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 -- 172.21.15.53:0/812791219 wait complete. 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 Processor -- start 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 -- start start 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.037+0000 7f88eb220700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88e410d330 con 0x7f88e4108dd0 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e8fbc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e8fbc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51524/0 (socket says 172.21.15.53:51524) 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e8fbc700 1 -- 172.21.15.53:0/759641397 learned_addr learned my addr 172.21.15.53:0/759641397 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e8fbc700 1 -- 172.21.15.53:0/759641397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e419d250 con 0x7f88e4108dd0 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e8fbc700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e419cd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009fb0 tx=0x7f88d8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e1ffb700 1 -- 172.21.15.53:0/759641397 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88d80036a0 con 0x7f88e4108dd0 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e1ffb700 1 -- 172.21.15.53:0/759641397 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88d801a430 con 0x7f88e4108dd0 2024-04-28T12:41:12.357 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e419d4e0 con 0x7f88e4108dd0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88e1ffb700 1 -- 172.21.15.53:0/759641397 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88d8019550 con 0x7f88e4108dd0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.038+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e41a01c0 con 0x7f88e4108dd0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.039+0000 7f88e1ffb700 1 -- 172.21.15.53:0/759641397 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f88d8003840 con 0x7f88e4108dd0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.039+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e4064a80 con 0x7f88e4108dd0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.039+0000 7f88e1ffb700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f88d40395b0 0x7f88d403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.040+0000 7f88e1ffb700 1 -- 172.21.15.53:0/759641397 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f88d804d980 con 0x7f88e4108dd0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.040+0000 7f88e3fff700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f88d40395b0 0x7f88d403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.040+0000 7f88e3fff700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f88d40395b0 0x7f88d403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88d0006fd0 tx=0x7f88d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.045+0000 7f88e1ffb700 1 -- 172.21.15.53:0/759641397 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f88d8003b30 con 0x7f88e4108dd0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.253+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f88e4105630 con 0x7f88d40395b0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.751+0000 7f88e1ffb700 1 -- 172.21.15.53:0/759641397 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f88e4105630 con 0x7f88d40395b0 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f88d40395b0 msgr2=0x7f88d403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:12.358 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f88d40395b0 0x7f88d403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88d0006fd0 tx=0x7f88d0006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 msgr2=0x7f88e419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e419cd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009fb0 tx=0x7f88d8011940 comp rx=0 tx=0).stop 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 shutdown_connections 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f88d40395b0 0x7f88d403ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 --2- 172.21.15.53:0/759641397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88e4108dd0 0x7f88e419cd10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 >> 172.21.15.53:0/759641397 conn(0x7f88e4104370 msgr2=0x7f88e4104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 shutdown_connections 2024-04-28T12:41:12.359 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:11.753+0000 7f88eb220700 1 -- 172.21.15.53:0/759641397 wait complete. 2024-04-28T12:41:12.973 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:12 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:12.973 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:12 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:13.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDw3vQU3JplBaj/sY50eKDX6yb/S8VrzD+EWWaUhoW7C2UShf4tolhLe7Y9cEEFd+ieJxWmOKmjy21slNSve2MYitgRXKNG9MixQaRoScpF08HwNedCK5SL+9SCCD7dhlmd//o1Fb6rs1WluhSs3a2RmWfvp4zWR4LNLm15Dp2ViBcHgIYeKl7MmKcm2M7Kk4XzIW+b3Ukdt/4hcQEJGyXggR4VEYuU63WeoZRismYs2pE8m8mSMwQVpUYDkgC00Y4RizxJ/ajps+FE7/IyXkPoFTGuHGCXjDpifxgESoHTv2XSguHdr6tePN2jhkQUXExfvN0JAt5AKLK3X22hJIEqucpNSDyymqlsZG8QLg+DyNWMldB4Uw1VAqAMeralEc9XajPKY0RvI6S2OS98oaWcrKuBljzBbQSdfvRwsv5+3Wbgvua5bhLqxnoHA9WZ89kU4sbFsDdi4DJk172olpUP7uiWe5bPP0c5sCsTqFFdBOAoYoIFFa6sT0YUtOe78Y0= ceph-10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.760+0000 7f0e38905700 1 Processor -- start 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.760+0000 7f0e38905700 1 -- start start 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e38905700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e341080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e38905700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e34108680 con 0x7f0e34107c90 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e341080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e341080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51528/0 (socket says 172.21.15.53:51528) 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3259c700 1 -- 172.21.15.53:0/175514276 learned_addr learned my addr 172.21.15.53:0/175514276 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3259c700 1 -- 172.21.15.53:0/175514276 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e34108e90 con 0x7f0e34107c90 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3259c700 1 --2- 172.21.15.53:0/175514276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e341080b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0e240098a0 tx=0x7f0e24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e66b20336c95b7a server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3159a700 1 -- 172.21.15.53:0/175514276 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e24004160 con 0x7f0e34107c90 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3159a700 1 -- 172.21.15.53:0/175514276 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0e24004300 con 0x7f0e34107c90 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.761+0000 7f0e3159a700 1 -- 172.21.15.53:0/175514276 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e24011af0 con 0x7f0e34107c90 2024-04-28T12:41:13.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.762+0000 7f0e38905700 1 -- 172.21.15.53:0/175514276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 msgr2=0x7f0e341080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.762+0000 7f0e38905700 1 --2- 172.21.15.53:0/175514276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e341080b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0e240098a0 tx=0x7f0e24009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.762+0000 7f0e38905700 1 -- 172.21.15.53:0/175514276 shutdown_connections 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.762+0000 7f0e38905700 1 --2- 172.21.15.53:0/175514276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e341080b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.762+0000 7f0e38905700 1 -- 172.21.15.53:0/175514276 >> 172.21.15.53:0/175514276 conn(0x7f0e341031f0 msgr2=0x7f0e34105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.762+0000 7f0e38905700 1 -- 172.21.15.53:0/175514276 shutdown_connections 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.762+0000 7f0e38905700 1 -- 172.21.15.53:0/175514276 wait complete. 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.763+0000 7f0e38905700 1 Processor -- start 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.763+0000 7f0e38905700 1 -- start start 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.763+0000 7f0e38905700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e3419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.763+0000 7f0e38905700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3410c240 con 0x7f0e34107c90 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.763+0000 7f0e3259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e3419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.763+0000 7f0e3259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e3419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51540/0 (socket says 172.21.15.53:51540) 2024-04-28T12:41:13.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.763+0000 7f0e3259c700 1 -- 172.21.15.53:0/2767636933 learned_addr learned my addr 172.21.15.53:0/2767636933 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.764+0000 7f0e3259c700 1 -- 172.21.15.53:0/2767636933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3419d280 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.764+0000 7f0e3259c700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e3419cd40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0e240117b0 tx=0x7f0e24011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.764+0000 7f0e237fe700 1 -- 172.21.15.53:0/2767636933 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e24003af0 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.764+0000 7f0e237fe700 1 -- 172.21.15.53:0/2767636933 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0e24020070 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.764+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3419d510 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.764+0000 7f0e237fe700 1 -- 172.21.15.53:0/2767636933 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e24019630 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.764+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e341a0200 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.765+0000 7f0e237fe700 1 -- 172.21.15.53:0/2767636933 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0e2401a430 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.765+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e34064a80 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.765+0000 7f0e237fe700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f0e1c039600 0x7f0e1c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.765+0000 7f0e237fe700 1 -- 172.21.15.53:0/2767636933 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0e2404db10 con 0x7f0e34107c90 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.765+0000 7f0e31d9b700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f0e1c039600 0x7f0e1c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.765+0000 7f0e31d9b700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f0e1c039600 0x7f0e1c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0e28006fd0 tx=0x7f0e28006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.770+0000 7f0e237fe700 1 -- 172.21.15.53:0/2767636933 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0e2400e260 con 0x7f0e34107c90 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.976+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f0e340008d0 con 0x7f0e1c039600 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.977+0000 7f0e237fe700 1 -- 172.21.15.53:0/2767636933 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f0e340008d0 con 0x7f0e1c039600 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.979+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f0e1c039600 msgr2=0x7f0e1c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.979+0000 7f0e38905700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f0e1c039600 0x7f0e1c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0e28006fd0 tx=0x7f0e28006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.979+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 msgr2=0x7f0e3419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.979+0000 7f0e38905700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e3419cd40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0e240117b0 tx=0x7f0e24011b70 comp rx=0 tx=0).stop 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.980+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 shutdown_connections 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.980+0000 7f0e38905700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f0e1c039600 0x7f0e1c03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.980+0000 7f0e38905700 1 --2- 172.21.15.53:0/2767636933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e34107c90 0x7f0e3419cd40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.980+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 >> 172.21.15.53:0/2767636933 conn(0x7f0e341031f0 msgr2=0x7f0e34104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.980+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 shutdown_connections 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:12.980+0000 7f0e38905700 1 -- 172.21.15.53:0/2767636933 wait complete. 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:Adding key to root@localhost authorized_keys... 2024-04-28T12:41:13.615 INFO:teuthology.orchestra.run.smithi053.stdout:Adding host smithi053... 2024-04-28T12:41:13.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:13 smithi053 ceph-mon[29182]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:14.869 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:14 smithi053 ceph-mon[29182]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:16.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:15 smithi053 ceph-mon[29182]: Deploying cephadm binary to smithi053 2024-04-28T12:41:19.193 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Added host 'smithi053' with addr '172.21.15.53' 2024-04-28T12:41:19.193 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.046+0000 7fd73675d700 1 Processor -- start 2024-04-28T12:41:19.193 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.046+0000 7fd73675d700 1 -- start start 2024-04-28T12:41:19.193 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.047+0000 7fd73675d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73010bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.047+0000 7fd73675d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd73007d6d0 con 0x7fd730109840 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.047+0000 7fd72ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73010bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.047+0000 7fd72ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73010bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51552/0 (socket says 172.21.15.53:51552) 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.047+0000 7fd72ffff700 1 -- 172.21.15.53:0/1311794627 learned_addr learned my addr 172.21.15.53:0/1311794627 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.047+0000 7fd72ffff700 1 -- 172.21.15.53:0/1311794627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73010c170 con 0x7fd730109840 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.047+0000 7fd72ffff700 1 --2- 172.21.15.53:0/1311794627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73010bc30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7200098a0 tx=0x7fd720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c55231b6194efd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd72effd700 1 -- 172.21.15.53:0/1311794627 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd720004160 con 0x7fd730109840 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd72effd700 1 -- 172.21.15.53:0/1311794627 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd720004300 con 0x7fd730109840 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd72effd700 1 -- 172.21.15.53:0/1311794627 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd720011af0 con 0x7fd730109840 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd73675d700 1 -- 172.21.15.53:0/1311794627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 msgr2=0x7fd73010bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd73675d700 1 --2- 172.21.15.53:0/1311794627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73010bc30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7200098a0 tx=0x7fd720009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd73675d700 1 -- 172.21.15.53:0/1311794627 shutdown_connections 2024-04-28T12:41:19.194 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd73675d700 1 --2- 172.21.15.53:0/1311794627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73010bc30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd73675d700 1 -- 172.21.15.53:0/1311794627 >> 172.21.15.53:0/1311794627 conn(0x7fd7301031d0 msgr2=0x7fd730105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd73675d700 1 -- 172.21.15.53:0/1311794627 shutdown_connections 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.048+0000 7fd73675d700 1 -- 172.21.15.53:0/1311794627 wait complete. 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.049+0000 7fd73675d700 1 Processor -- start 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.049+0000 7fd73675d700 1 -- start start 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.049+0000 7fd73675d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.049+0000 7fd73675d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd73010d0e0 con 0x7fd730109840 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51562/0 (socket says 172.21.15.53:51562) 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72ffff700 1 -- 172.21.15.53:0/546355287 learned_addr learned my addr 172.21.15.53:0/546355287 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72ffff700 1 -- 172.21.15.53:0/546355287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73019d1e0 con 0x7fd730109840 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72ffff700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73019cca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd720009fb0 tx=0x7fd720011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:19.195 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72d7fa700 1 -- 172.21.15.53:0/546355287 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7200036a0 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72d7fa700 1 -- 172.21.15.53:0/546355287 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd72001a430 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73019d470 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd72d7fa700 1 -- 172.21.15.53:0/546355287 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd720019550 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.050+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7301a0160 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.051+0000 7fd72d7fa700 1 -- 172.21.15.53:0/546355287 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd720019850 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.051+0000 7fd72d7fa700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fd718039560 0x7fd71803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.051+0000 7fd72d7fa700 1 -- 172.21.15.53:0/546355287 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd72004d940 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.051+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd730064a80 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.051+0000 7fd72f7fe700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fd718039560 0x7fd71803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.052+0000 7fd72f7fe700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fd718039560 0x7fd71803ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd724006fd0 tx=0x7fd724006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.056+0000 7fd72d7fa700 1 -- 172.21.15.53:0/546355287 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd72000bc40 con 0x7fd730109840 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:14.271+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}) -- 0x7fd730104430 con 0x7fd718039560 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.627+0000 7fd72d7fa700 1 -- 172.21.15.53:0/546355287 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fd730104430 con 0x7fd718039560 2024-04-28T12:41:19.196 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fd718039560 msgr2=0x7fd71803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fd718039560 0x7fd71803ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd724006fd0 tx=0x7fd724006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 msgr2=0x7fd73019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73019cca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd720009fb0 tx=0x7fd720011940 comp rx=0 tx=0).stop 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 shutdown_connections 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fd718039560 0x7fd71803ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 --2- 172.21.15.53:0/546355287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd730109840 0x7fd73019cca0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.629+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 >> 172.21.15.53:0/546355287 conn(0x7fd7301031d0 msgr2=0x7fd730103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.630+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 shutdown_connections 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:18.630+0000 7fd73675d700 1 -- 172.21.15.53:0/546355287 wait complete. 2024-04-28T12:41:19.197 INFO:teuthology.orchestra.run.smithi053.stdout:Deploying unmanaged mon service... 2024-04-28T12:41:19.882 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:19 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:19.882 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:19 smithi053 ceph-mon[29182]: Added host smithi053 2024-04-28T12:41:19.882 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:19 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:21.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:20 smithi053 ceph-mon[29182]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:21.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:20 smithi053 ceph-mon[29182]: Saving service mon spec with placement count:5 2024-04-28T12:41:21.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:20 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:21.507 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.631+0000 7efff7230700 1 Processor -- start 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.631+0000 7efff7230700 1 -- start start 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.631+0000 7efff7230700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff0108090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.631+0000 7efff7230700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efff0108660 con 0x7efff0107c70 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7efff4fcc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff0108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7efff4fcc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff0108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35214/0 (socket says 172.21.15.53:35214) 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7efff4fcc700 1 -- 172.21.15.53:0/2119940624 learned_addr learned my addr 172.21.15.53:0/2119940624 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7efff4fcc700 1 -- 172.21.15.53:0/2119940624 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff0108e70 con 0x7efff0107c70 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7efff4fcc700 1 --2- 172.21.15.53:0/2119940624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff0108090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7effe80098a0 tx=0x7effe8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d252b7281fa5d1ca server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7effef7fe700 1 -- 172.21.15.53:0/2119940624 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effe8004160 con 0x7efff0107c70 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7effef7fe700 1 -- 172.21.15.53:0/2119940624 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7effe8004300 con 0x7efff0107c70 2024-04-28T12:41:21.508 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.632+0000 7effef7fe700 1 -- 172.21.15.53:0/2119940624 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effe8011af0 con 0x7efff0107c70 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 -- 172.21.15.53:0/2119940624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 msgr2=0x7efff0108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 --2- 172.21.15.53:0/2119940624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff0108090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7effe80098a0 tx=0x7effe8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 -- 172.21.15.53:0/2119940624 shutdown_connections 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 --2- 172.21.15.53:0/2119940624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff0108090 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 -- 172.21.15.53:0/2119940624 >> 172.21.15.53:0/2119940624 conn(0x7efff0103230 msgr2=0x7efff0105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 -- 172.21.15.53:0/2119940624 shutdown_connections 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 -- 172.21.15.53:0/2119940624 wait complete. 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 Processor -- start 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.633+0000 7efff7230700 1 -- start start 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff7230700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff7230700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efff010c220 con 0x7efff0107c70 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff4fcc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:21.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff4fcc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff019ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35230/0 (socket says 172.21.15.53:35230) 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff4fcc700 1 -- 172.21.15.53:0/3646916565 learned_addr learned my addr 172.21.15.53:0/3646916565 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff4fcc700 1 -- 172.21.15.53:0/3646916565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff019d230 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff4fcc700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff019ccf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7effe80117b0 tx=0x7effe8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7effedffb700 1 -- 172.21.15.53:0/3646916565 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effe8003af0 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7effedffb700 1 -- 172.21.15.53:0/3646916565 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7effe8020070 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7effedffb700 1 -- 172.21.15.53:0/3646916565 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effe8019630 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.634+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff019d4c0 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.635+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff01a01a0 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.635+0000 7effedffb700 1 -- 172.21.15.53:0/3646916565 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7effe801a9c0 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.635+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff0064a80 con 0x7efff0107c70 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.636+0000 7effedffb700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7effdc039600 0x7effdc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:21.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.636+0000 7effedffb700 1 -- 172.21.15.53:0/3646916565 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7effe804dbf0 con 0x7efff0107c70 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.636+0000 7effeffff700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7effdc039600 0x7effdc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.636+0000 7effeffff700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7effdc039600 0x7effdc03bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7effe000ad30 tx=0x7effe00093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.641+0000 7effedffb700 1 -- 172.21.15.53:0/3646916565 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7effe800e260 con 0x7efff0107c70 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.868+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7efff00008d0 con 0x7effdc039600 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.873+0000 7effedffb700 1 -- 172.21.15.53:0/3646916565 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7efff00008d0 con 0x7effdc039600 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7effdc039600 msgr2=0x7effdc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7effdc039600 0x7effdc03bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7effe000ad30 tx=0x7effe00093f0 comp rx=0 tx=0).stop 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 msgr2=0x7efff019ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff019ccf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7effe80117b0 tx=0x7effe8011b70 comp rx=0 tx=0).stop 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 shutdown_connections 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7effdc039600 0x7effdc03bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 --2- 172.21.15.53:0/3646916565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efff0107c70 0x7efff019ccf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 >> 172.21.15.53:0/3646916565 conn(0x7efff0103230 msgr2=0x7efff01040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:21.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 shutdown_connections 2024-04-28T12:41:21.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:19.875+0000 7efff7230700 1 -- 172.21.15.53:0/3646916565 wait complete. 2024-04-28T12:41:21.512 INFO:teuthology.orchestra.run.smithi053.stdout:Deploying unmanaged mgr service... 2024-04-28T12:41:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:21 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:21 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:22.814 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.928+0000 7f58d77fd700 1 Processor -- start 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.928+0000 7f58d77fd700 1 -- start start 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.928+0000 7f58d77fd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d0107a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.928+0000 7f58d77fd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f58d007d690 con 0x7f58d0107670 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.928+0000 7f58d5599700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d0107a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.928+0000 7f58d5599700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d0107a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35244/0 (socket says 172.21.15.53:35244) 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.928+0000 7f58d5599700 1 -- 172.21.15.53:0/3771243434 learned_addr learned my addr 172.21.15.53:0/3771243434 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.929+0000 7f58d5599700 1 -- 172.21.15.53:0/3771243434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d0107fd0 con 0x7f58d0107670 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.929+0000 7f58d5599700 1 --2- 172.21.15.53:0/3771243434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d0107a90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58c00098a0 tx=0x7f58c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc5dfe8a8c90d9f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.929+0000 7f58c7fff700 1 -- 172.21.15.53:0/3771243434 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0004160 con 0x7f58d0107670 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.929+0000 7f58c7fff700 1 -- 172.21.15.53:0/3771243434 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58c0004300 con 0x7f58d0107670 2024-04-28T12:41:22.815 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.929+0000 7f58c7fff700 1 -- 172.21.15.53:0/3771243434 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0011a20 con 0x7f58d0107670 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.930+0000 7f58d77fd700 1 -- 172.21.15.53:0/3771243434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 msgr2=0x7f58d0107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.930+0000 7f58d77fd700 1 --2- 172.21.15.53:0/3771243434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d0107a90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f58c00098a0 tx=0x7f58c0009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.930+0000 7f58d77fd700 1 -- 172.21.15.53:0/3771243434 shutdown_connections 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.930+0000 7f58d77fd700 1 --2- 172.21.15.53:0/3771243434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d0107a90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.930+0000 7f58d77fd700 1 -- 172.21.15.53:0/3771243434 >> 172.21.15.53:0/3771243434 conn(0x7f58d0103230 msgr2=0x7f58d0105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.930+0000 7f58d77fd700 1 -- 172.21.15.53:0/3771243434 shutdown_connections 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.930+0000 7f58d77fd700 1 -- 172.21.15.53:0/3771243434 wait complete. 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d77fd700 1 Processor -- start 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d77fd700 1 -- start start 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d77fd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d77fd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f58d0109040 con 0x7f58d0107670 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d5599700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d5599700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d019ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35254/0 (socket says 172.21.15.53:35254) 2024-04-28T12:41:22.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d5599700 1 -- 172.21.15.53:0/3596215697 learned_addr learned my addr 172.21.15.53:0/3596215697 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d5599700 1 -- 172.21.15.53:0/3596215697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d019f3b0 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.931+0000 7f58d5599700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d019ee70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f58c00116e0 tx=0x7f58c0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.932+0000 7f58c67fc700 1 -- 172.21.15.53:0/3596215697 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0003b20 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.932+0000 7f58c67fc700 1 -- 172.21.15.53:0/3596215697 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58c001b8f0 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.932+0000 7f58c67fc700 1 -- 172.21.15.53:0/3596215697 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c00238c0 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.932+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d019f640 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.932+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d01a2330 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.933+0000 7f58c67fc700 1 -- 172.21.15.53:0/3596215697 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f58c001baa0 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.933+0000 7f58c67fc700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f58bc039560 0x7f58bc03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.933+0000 7f58c67fc700 1 -- 172.21.15.53:0/3596215697 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f58c004e050 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.933+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d0064a80 con 0x7f58d0107670 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.933+0000 7f58d4d98700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f58bc039560 0x7f58bc03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.933+0000 7f58d4d98700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f58bc039560 0x7f58bc03ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f58cc006fd0 tx=0x7f58cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:22.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:21.938+0000 7f58c67fc700 1 -- 172.21.15.53:0/3596215697 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f58c001f030 con 0x7f58d0107670 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.204+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f58d002e6d0 con 0x7f58bc039560 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.209+0000 7f58c67fc700 1 -- 172.21.15.53:0/3596215697 <== mgr.14120 v2:172.21.15.53:6800/2275002354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f58d002e6d0 con 0x7f58bc039560 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.211+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f58bc039560 msgr2=0x7f58bc03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.211+0000 7f58d77fd700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f58bc039560 0x7f58bc03ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f58cc006fd0 tx=0x7f58cc006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.211+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 msgr2=0x7f58d019ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.211+0000 7f58d77fd700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d019ee70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f58c00116e0 tx=0x7f58c0011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.212+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 shutdown_connections 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.212+0000 7f58d77fd700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f58bc039560 0x7f58bc03ba20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.212+0000 7f58d77fd700 1 --2- 172.21.15.53:0/3596215697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58d0107670 0x7f58d019ee70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.212+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 >> 172.21.15.53:0/3596215697 conn(0x7f58d0103230 msgr2=0x7f58d0195ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.212+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 shutdown_connections 2024-04-28T12:41:22.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:22.212+0000 7f58d77fd700 1 -- 172.21.15.53:0/3596215697 wait complete. 2024-04-28T12:41:23.444 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:23 smithi053 ceph-mon[29182]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:23.445 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:23 smithi053 ceph-mon[29182]: Saving service mgr spec with placement count:2 2024-04-28T12:41:23.445 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:23 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.277+0000 7f21dc37f700 1 Processor -- start 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.278+0000 7f21dc37f700 1 -- start start 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.278+0000 7f21dc37f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d41085b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.278+0000 7f21dc37f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f21d4108b80 con 0x7f21d4108190 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.278+0000 7f21da11b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d41085b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.278+0000 7f21da11b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d41085b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35258/0 (socket says 172.21.15.53:35258) 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.278+0000 7f21da11b700 1 -- 172.21.15.53:0/944875057 learned_addr learned my addr 172.21.15.53:0/944875057 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.279+0000 7f21da11b700 1 -- 172.21.15.53:0/944875057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d4108eb0 con 0x7f21d4108190 2024-04-28T12:41:24.088 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.279+0000 7f21da11b700 1 --2- 172.21.15.53:0/944875057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d41085b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f21d00098a0 tx=0x7f21d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5391bf743d1b517f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.279+0000 7f21d9119700 1 -- 172.21.15.53:0/944875057 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21d0004160 con 0x7f21d4108190 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.279+0000 7f21d9119700 1 -- 172.21.15.53:0/944875057 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21d0004300 con 0x7f21d4108190 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.279+0000 7f21d9119700 1 -- 172.21.15.53:0/944875057 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21d0011a20 con 0x7f21d4108190 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 -- 172.21.15.53:0/944875057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 msgr2=0x7f21d41085b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 --2- 172.21.15.53:0/944875057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d41085b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f21d00098a0 tx=0x7f21d0009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 -- 172.21.15.53:0/944875057 shutdown_connections 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 --2- 172.21.15.53:0/944875057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d41085b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 -- 172.21.15.53:0/944875057 >> 172.21.15.53:0/944875057 conn(0x7f21d4102970 msgr2=0x7f21d4104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 -- 172.21.15.53:0/944875057 shutdown_connections 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 -- 172.21.15.53:0/944875057 wait complete. 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 Processor -- start 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.280+0000 7f21dc37f700 1 -- start start 2024-04-28T12:41:24.089 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21dc37f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21dc37f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410c260 con 0x7f21d4108190 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21da11b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21da11b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35260/0 (socket says 172.21.15.53:35260) 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21da11b700 1 -- 172.21.15.53:0/2840008834 learned_addr learned my addr 172.21.15.53:0/2840008834 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21da11b700 1 -- 172.21.15.53:0/2840008834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d419d200 con 0x7f21d4108190 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21da11b700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d419ccc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21d00116e0 tx=0x7f21d0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21cb7fe700 1 -- 172.21.15.53:0/2840008834 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21d0003b20 con 0x7f21d4108190 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21cb7fe700 1 -- 172.21.15.53:0/2840008834 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21d001b8f0 con 0x7f21d4108190 2024-04-28T12:41:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d419d490 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21cb7fe700 1 -- 172.21.15.53:0/2840008834 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21d00238c0 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.281+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d41a0180 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.282+0000 7f21cb7fe700 1 -- 172.21.15.53:0/2840008834 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f21d0023b70 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.282+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d4064a80 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.282+0000 7f21cb7fe700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f21c0039560 0x7f21c003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.282+0000 7f21cb7fe700 1 -- 172.21.15.53:0/2840008834 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f21d004e070 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.283+0000 7f21d991a700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f21c0039560 0x7f21c003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.283+0000 7f21d991a700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f21c0039560 0x7f21c003ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f21c4006fd0 tx=0x7f21c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.288+0000 7f21cb7fe700 1 -- 172.21.15.53:0/2840008834 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f21d001f030 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.498+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f21d402e6e0 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.501+0000 7f21cb7fe700 1 -- 172.21.15.53:0/2840008834 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f21d0020d70 con 0x7f21d4108190 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f21c0039560 msgr2=0x7f21c003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f21c0039560 0x7f21c003ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f21c4006fd0 tx=0x7f21c4006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:24.091 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 msgr2=0x7f21d419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:24.092 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d419ccc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21d00116e0 tx=0x7f21d0011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.092 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 shutdown_connections 2024-04-28T12:41:24.092 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f21c0039560 0x7f21c003ba20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.092 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 --2- 172.21.15.53:0/2840008834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d4108190 0x7f21d419ccc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.092 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 >> 172.21.15.53:0/2840008834 conn(0x7f21d4102970 msgr2=0x7f21d4193880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:24.092 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 shutdown_connections 2024-04-28T12:41:24.092 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:23.503+0000 7f21dc37f700 1 -- 172.21.15.53:0/2840008834 wait complete. 2024-04-28T12:41:24.859 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:24 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2840008834' entity='client.admin' 2024-04-28T12:41:25.478 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.639+0000 7f0a19e98700 1 Processor -- start 2024-04-28T12:41:25.478 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.639+0000 7f0a19e98700 1 -- start start 2024-04-28T12:41:25.478 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.639+0000 7f0a19e98700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a141080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.639+0000 7f0a19e98700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0a14108680 con 0x7f0a14107c90 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a141080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a141080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35268/0 (socket says 172.21.15.53:35268) 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a137fe700 1 -- 172.21.15.53:0/497095006 learned_addr learned my addr 172.21.15.53:0/497095006 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a137fe700 1 -- 172.21.15.53:0/497095006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a14108e90 con 0x7f0a14107c90 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a137fe700 1 --2- 172.21.15.53:0/497095006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a141080b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a040098a0 tx=0x7f0a04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43e633066cdf821 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a127fc700 1 -- 172.21.15.53:0/497095006 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a04004160 con 0x7f0a14107c90 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a127fc700 1 -- 172.21.15.53:0/497095006 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a04004300 con 0x7f0a14107c90 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.640+0000 7f0a127fc700 1 -- 172.21.15.53:0/497095006 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a04011a20 con 0x7f0a14107c90 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.641+0000 7f0a19e98700 1 -- 172.21.15.53:0/497095006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 msgr2=0x7f0a141080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.641+0000 7f0a19e98700 1 --2- 172.21.15.53:0/497095006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a141080b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a040098a0 tx=0x7f0a04009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.641+0000 7f0a19e98700 1 -- 172.21.15.53:0/497095006 shutdown_connections 2024-04-28T12:41:25.479 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.641+0000 7f0a19e98700 1 --2- 172.21.15.53:0/497095006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a141080b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.641+0000 7f0a19e98700 1 -- 172.21.15.53:0/497095006 >> 172.21.15.53:0/497095006 conn(0x7f0a141031f0 msgr2=0x7f0a14105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.641+0000 7f0a19e98700 1 -- 172.21.15.53:0/497095006 shutdown_connections 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.641+0000 7f0a19e98700 1 -- 172.21.15.53:0/497095006 wait complete. 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a19e98700 1 Processor -- start 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a19e98700 1 -- start start 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a19e98700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a1407ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a19e98700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1410c240 con 0x7f0a14107c90 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a1407ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a1407ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35270/0 (socket says 172.21.15.53:35270) 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a137fe700 1 -- 172.21.15.53:0/1795494218 learned_addr learned my addr 172.21.15.53:0/1795494218 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.642+0000 7f0a137fe700 1 -- 172.21.15.53:0/1795494218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1419fa30 con 0x7f0a14107c90 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.643+0000 7f0a137fe700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a1407ca40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a04000c00 tx=0x7f0a04011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:25.480 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.643+0000 7f0a10ff9700 1 -- 172.21.15.53:0/1795494218 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a04003aa0 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.643+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1419fc00 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.643+0000 7f0a10ff9700 1 -- 172.21.15.53:0/1795494218 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a0401b8f0 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.643+0000 7f0a10ff9700 1 -- 172.21.15.53:0/1795494218 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a04023890 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.643+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1419ffa0 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.644+0000 7f0a10ff9700 1 -- 172.21.15.53:0/1795494218 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0a0402c3f0 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.644+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a14064a80 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.644+0000 7f0a10ff9700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f09fc0395b0 0x7f09fc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.644+0000 7f0a10ff9700 1 -- 172.21.15.53:0/1795494218 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0a0404de60 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.644+0000 7f0a12ffd700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f09fc0395b0 0x7f09fc03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.645+0000 7f0a12ffd700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f09fc0395b0 0x7f09fc03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a08006fd0 tx=0x7f0a08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.649+0000 7f0a10ff9700 1 -- 172.21.15.53:0/1795494218 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0a0401f030 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.863+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f0a141a0220 con 0x7f0a14107c90 2024-04-28T12:41:25.481 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.865+0000 7f0a10ff9700 1 -- 172.21.15.53:0/1795494218 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f0a040292a0 con 0x7f0a14107c90 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f09fc0395b0 msgr2=0x7f09fc03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f09fc0395b0 0x7f09fc03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a08006fd0 tx=0x7f0a08006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 msgr2=0x7f0a1407ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a1407ca40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a04000c00 tx=0x7f0a04011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 shutdown_connections 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f09fc0395b0 0x7f09fc03ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 --2- 172.21.15.53:0/1795494218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a14107c90 0x7f0a1407ca40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 >> 172.21.15.53:0/1795494218 conn(0x7f0a141031f0 msgr2=0x7f0a14103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.868+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 shutdown_connections 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:24.869+0000 7f0a19e98700 1 -- 172.21.15.53:0/1795494218 wait complete. 2024-04-28T12:41:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling the dashboard module... 2024-04-28T12:41:26.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:25 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1795494218' entity='client.admin' 2024-04-28T12:41:26.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:25 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:27.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-mon[29182]: from='mgr.14120 172.21.15.53:0/705603645' entity='mgr.x' 2024-04-28T12:41:27.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2085656473' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-28T12:41:27.395 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: ignoring --setuser ceph since I am not root 2024-04-28T12:41:27.396 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: ignoring --setgroup ceph since I am not root 2024-04-28T12:41:27.396 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:27.362+0000 7f85b33c2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:41:27.895 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:27.552+0000 7f85b33c2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:27.895 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:27.661+0000 7f85b33c2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.969+0000 7f7539141700 1 Processor -- start 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.969+0000 7f7539141700 1 -- start start 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.969+0000 7f7539141700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f75341080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.969+0000 7f7539141700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7534108680 con 0x7f7534107c90 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.969+0000 7f7532d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f75341080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.970+0000 7f7532d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f75341080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35286/0 (socket says 172.21.15.53:35286) 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.970+0000 7f7532d9d700 1 -- 172.21.15.53:0/3917523346 learned_addr learned my addr 172.21.15.53:0/3917523346 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.970+0000 7f7532d9d700 1 -- 172.21.15.53:0/3917523346 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7534108e90 con 0x7f7534107c90 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.970+0000 7f7532d9d700 1 --2- 172.21.15.53:0/3917523346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f75341080b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f752400bb40 tx=0x7f752400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd5e8366ba9b48b server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.970+0000 7f7531d9b700 1 -- 172.21.15.53:0/3917523346 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f752400dbe0 con 0x7f7534107c90 2024-04-28T12:41:27.940 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.970+0000 7f7531d9b700 1 -- 172.21.15.53:0/3917523346 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75240044e0 con 0x7f7534107c90 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.970+0000 7f7531d9b700 1 -- 172.21.15.53:0/3917523346 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7524010470 con 0x7f7534107c90 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.971+0000 7f7539141700 1 -- 172.21.15.53:0/3917523346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 msgr2=0x7f75341080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.971+0000 7f7539141700 1 --2- 172.21.15.53:0/3917523346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f75341080b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f752400bb40 tx=0x7f752400be50 comp rx=0 tx=0).stop 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.971+0000 7f7539141700 1 -- 172.21.15.53:0/3917523346 shutdown_connections 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.971+0000 7f7539141700 1 --2- 172.21.15.53:0/3917523346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f75341080b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.971+0000 7f7539141700 1 -- 172.21.15.53:0/3917523346 >> 172.21.15.53:0/3917523346 conn(0x7f75341031f0 msgr2=0x7f7534105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.971+0000 7f7539141700 1 -- 172.21.15.53:0/3917523346 shutdown_connections 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.971+0000 7f7539141700 1 -- 172.21.15.53:0/3917523346 wait complete. 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7539141700 1 Processor -- start 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7539141700 1 -- start start 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7539141700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f753419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7539141700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f753407bfc0 con 0x7f7534107c90 2024-04-28T12:41:27.941 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7532d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f753419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7532d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f753419eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35298/0 (socket says 172.21.15.53:35298) 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7532d9d700 1 -- 172.21.15.53:0/2085656473 learned_addr learned my addr 172.21.15.53:0/2085656473 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.972+0000 7f7532d9d700 1 -- 172.21.15.53:0/2085656473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753419f400 con 0x7f7534107c90 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.973+0000 7f7532d9d700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f753419eec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7524003c30 tx=0x7f7524003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.973+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7524010720 con 0x7f7534107c90 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.973+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f752401d070 con 0x7f7534107c90 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.973+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753419f690 con 0x7f7534107c90 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.973+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75341a2380 con 0x7f7534107c90 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.973+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7524021aa0 con 0x7f7534107c90 2024-04-28T12:41:27.942 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.974+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f752401f030 con 0x7f7534107c90 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.974+0000 7f751bfff700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f751c039600 0x7f751c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.974+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f752404e870 con 0x7f7534107c90 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.974+0000 7f753259c700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f751c039600 0x7f751c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.974+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7520005360 con 0x7f7534107c90 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.975+0000 7f753259c700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f751c039600 0x7f751c03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7528006fd0 tx=0x7f7528006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:25.979+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7524015070 con 0x7f7534107c90 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:26.253+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f7520005650 con 0x7f7534107c90 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.177+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f7524026210 con 0x7f7534107c90 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.178+0000 7f751bfff700 1 -- 172.21.15.53:0/2085656473 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f7524019770 con 0x7f7534107c90 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f751c039600 msgr2=0x7f751c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f751c039600 0x7f751c03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7528006fd0 tx=0x7f7528006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 msgr2=0x7f753419eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:27.943 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f753419eec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7524003c30 tx=0x7f7524003ff0 comp rx=0 tx=0).stop 2024-04-28T12:41:27.944 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 shutdown_connections 2024-04-28T12:41:27.944 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f751c039600 0x7f751c03bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:27.944 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 --2- 172.21.15.53:0/2085656473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7534107c90 0x7f753419eec0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:27.944 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 >> 172.21.15.53:0/2085656473 conn(0x7f75341031f0 msgr2=0x7f75341910d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:27.944 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 shutdown_connections 2024-04-28T12:41:27.944 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:27.180+0000 7f7539141700 1 -- 172.21.15.53:0/2085656473 wait complete. 2024-04-28T12:41:28.189 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:28 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2085656473' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-28T12:41:28.189 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:28 smithi053 ceph-mon[29182]: mgrmap e9: x(active, since 20s) 2024-04-28T12:41:28.190 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:27.988+0000 7f85b33c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:41:28.190 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:28 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:28.077+0000 7f85b33c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:41:29.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb8a7f700 1 Processor -- start 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb8a7f700 1 -- start start 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb8a7f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb410bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb8a7f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fabb407d620 con 0x7fabb41098c0 2024-04-28T12:41:29.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb410bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb410bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48566/0 (socket says 172.21.15.53:48566) 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb259c700 1 -- 172.21.15.53:0/39249142 learned_addr learned my addr 172.21.15.53:0/39249142 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.390+0000 7fabb259c700 1 -- 172.21.15.53:0/39249142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb410c1f0 con 0x7fabb41098c0 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.391+0000 7fabb259c700 1 --2- 172.21.15.53:0/39249142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb410bcb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faba40098a0 tx=0x7faba4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845d8943a0498b54 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.391+0000 7fabb159a700 1 -- 172.21.15.53:0/39249142 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faba4004160 con 0x7fabb41098c0 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.391+0000 7fabb159a700 1 -- 172.21.15.53:0/39249142 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba4004300 con 0x7fabb41098c0 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.391+0000 7fabb159a700 1 -- 172.21.15.53:0/39249142 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faba4011a20 con 0x7fabb41098c0 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.391+0000 7fabb8a7f700 1 -- 172.21.15.53:0/39249142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 msgr2=0x7fabb410bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.391+0000 7fabb8a7f700 1 --2- 172.21.15.53:0/39249142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb410bcb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faba40098a0 tx=0x7faba4009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.392+0000 7fabb8a7f700 1 -- 172.21.15.53:0/39249142 shutdown_connections 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.392+0000 7fabb8a7f700 1 --2- 172.21.15.53:0/39249142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb410bcb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.392+0000 7fabb8a7f700 1 -- 172.21.15.53:0/39249142 >> 172.21.15.53:0/39249142 conn(0x7fabb4103230 msgr2=0x7fabb4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:29.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.392+0000 7fabb8a7f700 1 -- 172.21.15.53:0/39249142 shutdown_connections 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.392+0000 7fabb8a7f700 1 -- 172.21.15.53:0/39249142 wait complete. 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.392+0000 7fabb8a7f700 1 Processor -- start 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.392+0000 7fabb8a7f700 1 -- start start 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb8a7f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb4109140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb8a7f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fabb410d130 con 0x7fabb41098c0 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb4109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb4109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48580/0 (socket says 172.21.15.53:48580) 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb259c700 1 -- 172.21.15.53:0/2309786101 learned_addr learned my addr 172.21.15.53:0/2309786101 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb259c700 1 -- 172.21.15.53:0/2309786101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb4109680 con 0x7fabb41098c0 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb259c700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb4109140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faba40116e0 tx=0x7faba4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7faba37fe700 1 -- 172.21.15.53:0/2309786101 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faba4003b20 con 0x7fabb41098c0 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7faba37fe700 1 -- 172.21.15.53:0/2309786101 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba401b8f0 con 0x7fabb41098c0 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb4107870 con 0x7fabb41098c0 2024-04-28T12:41:29.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.393+0000 7faba37fe700 1 -- 172.21.15.53:0/2309786101 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faba40238c0 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.394+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb4107db0 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.394+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb4064a80 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.395+0000 7faba37fe700 1 -- 172.21.15.53:0/2309786101 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7faba401baa0 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.395+0000 7faba37fe700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 0x7fab9c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.395+0000 7faba37fe700 1 -- 172.21.15.53:0/2309786101 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faba404e090 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.395+0000 7fabb1d9b700 1 -- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 msgr2=0x7fab9c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2275002354 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.395+0000 7fabb1d9b700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 0x7fab9c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.400+0000 7faba37fe700 1 -- 172.21.15.53:0/2309786101 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faba401f030 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.596+0000 7fabb1d9b700 1 -- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 msgr2=0x7fab9c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2275002354 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.596+0000 7fabb1d9b700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 0x7fab9c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.716+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fabb41080f0 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.717+0000 7faba37fe700 1 -- 172.21.15.53:0/2309786101 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fabb41080f0 con 0x7fabb41098c0 2024-04-28T12:41:29.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 msgr2=0x7fab9c03ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 0x7fab9c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 msgr2=0x7fabb4109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb4109140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faba40116e0 tx=0x7faba4011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 shutdown_connections 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7fab9c0395b0 0x7fab9c03ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 --2- 172.21.15.53:0/2309786101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb41098c0 0x7fabb4109140 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 >> 172.21.15.53:0/2309786101 conn(0x7fabb4103230 msgr2=0x7fabb4103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 shutdown_connections 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:28.719+0000 7fabb8a7f700 1 -- 172.21.15.53:0/2309786101 wait complete. 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for the mgr to restart... 2024-04-28T12:41:29.294 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr epoch 9... 2024-04-28T12:41:29.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:29 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2309786101' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-28T12:41:29.645 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:29 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:29.350+0000 7f85b33c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:41:29.645 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:29 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:29.531+0000 7f85b33c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:30.011 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:29 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:29.714+0000 7f85b33c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:41:30.011 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:30 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:30.015+0000 7f85b33c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:41:30.281 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:30 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:30.115+0000 7f85b33c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:41:30.536 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:30 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:30.331+0000 7f85b33c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:41:30.895 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:30 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:30.540+0000 7f85b33c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:31.395 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:31 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:31.180+0000 7f85b33c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:41:31.895 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:31 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:31.623+0000 7f85b33c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:41:31.895 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:31 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:31.725+0000 7f85b33c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:41:32.395 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:31 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:31.936+0000 7f85b33c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:41:32.831 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:32 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:32.484+0000 7f85b33c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:41:32.831 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:32 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:32.580+0000 7f85b33c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:41:33.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:32 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:32.834+0000 7f85b33c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:41:33.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:32 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:32.974+0000 7f85b33c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:41:33.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:33.074+0000 7f85b33c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:41:33.504 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:33.195+0000 7f85b33c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:33.504 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:33.508+0000 7f85b33c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:41:34.395 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:34.139+0000 7f85b33c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:41:34.395 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:41:34.365+0000 7f85b33c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:41:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: Active manager daemon x restarted 2024-04-28T12:41:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: Activating manager daemon x 2024-04-28T12:41:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: osdmap e3: 0 total, 0 up, 0 in 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: mgrmap e10: x(active, starting, since 0.00327592s) 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: Manager daemon x is now available 2024-04-28T12:41:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:35.597 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T12:41:35.597 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T12:41:35.597 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:35 smithi053 ceph-mon[29182]: mgrmap e11: x(active, since 1.00662s) 2024-04-28T12:41:36.333 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:41:36.333 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-28T12:41:36.333 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-28T12:41:36.333 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.677+0000 7f82ddad5700 1 Processor -- start 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.677+0000 7f82ddad5700 1 -- start start 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.677+0000 7f82ddad5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d810bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.677+0000 7f82ddad5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f82d807cea0 con 0x7f82d8109800 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d810bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d810bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48582/0 (socket says 172.21.15.53:48582) 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d77fe700 1 -- 172.21.15.53:0/3923167990 learned_addr learned my addr 172.21.15.53:0/3923167990 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d77fe700 1 -- 172.21.15.53:0/3923167990 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d810c130 con 0x7f82d8109800 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d77fe700 1 --2- 172.21.15.53:0/3923167990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d810bbf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82c80098a0 tx=0x7f82c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea784a6f54e1df5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d67fc700 1 -- 172.21.15.53:0/3923167990 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c8004160 con 0x7f82d8109800 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d67fc700 1 -- 172.21.15.53:0/3923167990 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82c8004300 con 0x7f82d8109800 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.678+0000 7f82d67fc700 1 -- 172.21.15.53:0/3923167990 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c8011a20 con 0x7f82d8109800 2024-04-28T12:41:36.334 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.679+0000 7f82ddad5700 1 -- 172.21.15.53:0/3923167990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 msgr2=0x7f82d810bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.679+0000 7f82ddad5700 1 --2- 172.21.15.53:0/3923167990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d810bbf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82c80098a0 tx=0x7f82c8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.679+0000 7f82ddad5700 1 -- 172.21.15.53:0/3923167990 shutdown_connections 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.679+0000 7f82ddad5700 1 --2- 172.21.15.53:0/3923167990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d810bbf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.679+0000 7f82ddad5700 1 -- 172.21.15.53:0/3923167990 >> 172.21.15.53:0/3923167990 conn(0x7f82d81031f0 msgr2=0x7f82d8105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.679+0000 7f82ddad5700 1 -- 172.21.15.53:0/3923167990 shutdown_connections 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.679+0000 7f82ddad5700 1 -- 172.21.15.53:0/3923167990 wait complete. 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.680+0000 7f82ddad5700 1 Processor -- start 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.680+0000 7f82ddad5700 1 -- start start 2024-04-28T12:41:36.335 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.680+0000 7f82ddad5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82ddad5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f82d810d140 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48584/0 (socket says 172.21.15.53:48584) 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d77fe700 1 -- 172.21.15.53:0/162592669 learned_addr learned my addr 172.21.15.53:0/162592669 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d77fe700 1 -- 172.21.15.53:0/162592669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d819d1e0 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d77fe700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d819cca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f82c8004660 tx=0x7f82c8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c8003aa0 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82c801b8f0 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d819d470 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c8023900 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.681+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d81a0160 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.682+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f82c8023bc0 con 0x7f82d8109800 2024-04-28T12:41:36.336 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.682+0000 7f82d4ff9700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.682+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 --> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f82c003c1d0 con 0x7f82c0039600 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.683+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f82c804e0d0 con 0x7f82d8109800 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.683+0000 7f82d6ffd700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 msgr2=0x7f82c003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2275002354 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.683+0000 7f82d6ffd700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.884+0000 7f82d6ffd700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 msgr2=0x7f82c003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2275002354 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:29.884+0000 7f82d6ffd700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:30.285+0000 7f82d6ffd700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 msgr2=0x7f82c003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2275002354 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:30.285+0000 7f82d6ffd700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:31.086+0000 7f82d6ffd700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 msgr2=0x7f82c003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2275002354 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:31.086+0000 7f82d6ffd700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:32.687+0000 7f82d6ffd700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 msgr2=0x7f82c003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2275002354 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:32.687+0000 7f82d6ffd700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-28T12:41:36.337 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:34.369+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f82c802c3f0 con 0x7f82d8109800 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:34.369+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 msgr2=0x7f82c003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:34.369+0000 7f82d4ff9700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.370+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f82c80178f0 con 0x7f82d8109800 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.371+0000 7f82d4ff9700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f82c003cfe0 0x7f82c003f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.372+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f82c003c1d0 con 0x7f82c003cfe0 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.372+0000 7f82d6ffd700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f82c003cfe0 0x7f82c003f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.372+0000 7f82d6ffd700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f82c003cfe0 0x7f82c003f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f82cc003a10 tx=0x7f82cc0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.373+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f82c003c1d0 con 0x7f82c003cfe0 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.381+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f82d806a540 con 0x7f82c003cfe0 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82d4ff9700 1 -- 172.21.15.53:0/162592669 <== mgr.14152 v2:172.21.15.53:6800/3118663054 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f82d806a540 con 0x7f82c003cfe0 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f82c003cfe0 msgr2=0x7f82c003f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f82c003cfe0 0x7f82c003f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f82cc003a10 tx=0x7f82cc0092b0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 msgr2=0x7f82d819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:36.338 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d819cca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f82c8004660 tx=0x7f82c8011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 shutdown_connections 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f82c003cfe0 0x7f82c003f3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82d8109800 0x7f82d819cca0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 --2- 172.21.15.53:0/162592669 >> [v2:172.21.15.53:6800/2275002354,v1:172.21.15.53:6801/2275002354] conn(0x7f82c0039600 0x7f82c003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 >> 172.21.15.53:0/162592669 conn(0x7f82d81031f0 msgr2=0x7f82d8103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 shutdown_connections 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:35.382+0000 7f82ddad5700 1 -- 172.21.15.53:0/162592669 wait complete. 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:mgr epoch 9 is available 2024-04-28T12:41:36.339 INFO:teuthology.orchestra.run.smithi053.stdout:Generating a dashboard self-signed certificate... 2024-04-28T12:41:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:36 smithi053 ceph-mon[29182]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-28T12:41:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:36 smithi053 ceph-mon[29182]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-28T12:41:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:37.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-28T12:41:37.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d6c18b700 1 Processor -- start 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d6c18b700 1 -- start start 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d6c18b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d641091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d6c18b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0d641097c0 con 0x7f0d64108dd0 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d69f27700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d641091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d69f27700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d641091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37984/0 (socket says 172.21.15.53:37984) 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d69f27700 1 -- 172.21.15.53:0/224903039 learned_addr learned my addr 172.21.15.53:0/224903039 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.790+0000 7f0d69f27700 1 -- 172.21.15.53:0/224903039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d64109f80 con 0x7f0d64108dd0 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d69f27700 1 --2- 172.21.15.53:0/224903039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d641091f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098a0 tx=0x7f0d54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a96c4b75671a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d68f25700 1 -- 172.21.15.53:0/224903039 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d54004160 con 0x7f0d64108dd0 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d68f25700 1 -- 172.21.15.53:0/224903039 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d54004300 con 0x7f0d64108dd0 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d68f25700 1 -- 172.21.15.53:0/224903039 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d54011a20 con 0x7f0d64108dd0 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d6c18b700 1 -- 172.21.15.53:0/224903039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 msgr2=0x7f0d641091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:37.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d6c18b700 1 --2- 172.21.15.53:0/224903039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d641091f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098a0 tx=0x7f0d54009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d6c18b700 1 -- 172.21.15.53:0/224903039 shutdown_connections 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d6c18b700 1 --2- 172.21.15.53:0/224903039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d641091f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d6c18b700 1 -- 172.21.15.53:0/224903039 >> 172.21.15.53:0/224903039 conn(0x7f0d64104370 msgr2=0x7f0d641067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.791+0000 7f0d6c18b700 1 -- 172.21.15.53:0/224903039 shutdown_connections 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d6c18b700 1 -- 172.21.15.53:0/224903039 wait complete. 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d6c18b700 1 Processor -- start 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d6c18b700 1 -- start start 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d6c18b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d6407f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d6c18b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6410d330 con 0x7f0d64108dd0 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d69f27700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d6407f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d69f27700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d6407f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38000/0 (socket says 172.21.15.53:38000) 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.792+0000 7f0d69f27700 1 -- 172.21.15.53:0/3677791647 learned_addr learned my addr 172.21.15.53:0/3677791647 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:37.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.793+0000 7f0d69f27700 1 -- 172.21.15.53:0/3677791647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6407f930 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.793+0000 7f0d69f27700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d6407f3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0d540119f0 tx=0x7f0d54011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.793+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d540039e0 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.793+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d5401b8f0 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.793+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d6407db20 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.793+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d540237a0 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.793+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d6407e060 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.794+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f0d5402c3f0 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.794+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d64064a80 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.794+0000 7f0d5affd700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0d500394f0 0x7f0d5003b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.794+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0d5404df50 con 0x7f0d64108dd0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.794+0000 7f0d69726700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0d500394f0 0x7f0d5003b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.795+0000 7f0d69726700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0d500394f0 0x7f0d5003b9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d60006fd0 tx=0x7f0d60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.799+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0d5401f030 con 0x7f0d64108dd0 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:36.929+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f0d540193d0 con 0x7f0d64108dd0 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.018+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f0d6407e970 con 0x7f0d500394f0 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.109+0000 7f0d5affd700 1 -- 172.21.15.53:0/3677791647 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f0d6407e970 con 0x7f0d500394f0 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0d500394f0 msgr2=0x7f0d5003b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0d500394f0 0x7f0d5003b9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d60006fd0 tx=0x7f0d60006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 msgr2=0x7f0d6407f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d6407f3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0d540119f0 tx=0x7f0d54011ad0 comp rx=0 tx=0).stop 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 shutdown_connections 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0d500394f0 0x7f0d5003b9b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 --2- 172.21.15.53:0/3677791647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d64108dd0 0x7f0d6407f3f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 >> 172.21.15.53:0/3677791647 conn(0x7f0d64104370 msgr2=0x7f0d64104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 shutdown_connections 2024-04-28T12:41:37.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:37.111+0000 7f0d6c18b700 1 -- 172.21.15.53:0/3677791647 wait complete. 2024-04-28T12:41:37.616 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial admin user... 2024-04-28T12:41:38.056 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:36] ENGINE Bus STARTING 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:36] ENGINE Serving on https://172.21.15.53:7150 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:36] ENGINE Serving on http://172.21.15.53:8765 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: [28/Apr/2024:12:41:36] ENGINE Bus STARTED 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: mgrmap e12: x(active, since 2s) 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:38.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SLzjnXXLP3kT6hHncqWcJOyWiZHmBtG3pNryksetF0EIK4H7oz/Nq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714308098, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf9019700 1 Processor -- start 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf9019700 1 -- start start 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf9019700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf407db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf9019700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf407e0c0 con 0x7f3cf407f680 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf407db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf407db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38012/0 (socket says 172.21.15.53:38012) 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf2d9d700 1 -- 172.21.15.53:0/2149580570 learned_addr learned my addr 172.21.15.53:0/2149580570 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.085+0000 7f3cf2d9d700 1 -- 172.21.15.53:0/2149580570 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf407e240 con 0x7f3cf407f680 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf2d9d700 1 --2- 172.21.15.53:0/2149580570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf407db80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80098a0 tx=0x7f3ce8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e841468e89fd1110 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf1d9b700 1 -- 172.21.15.53:0/2149580570 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce8004160 con 0x7f3cf407f680 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf1d9b700 1 -- 172.21.15.53:0/2149580570 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ce8004300 con 0x7f3cf407f680 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf1d9b700 1 -- 172.21.15.53:0/2149580570 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce8011a20 con 0x7f3cf407f680 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf9019700 1 -- 172.21.15.53:0/2149580570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 msgr2=0x7f3cf407db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf9019700 1 --2- 172.21.15.53:0/2149580570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf407db80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80098a0 tx=0x7f3ce8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf9019700 1 -- 172.21.15.53:0/2149580570 shutdown_connections 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf9019700 1 --2- 172.21.15.53:0/2149580570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf407db80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.086+0000 7f3cf9019700 1 -- 172.21.15.53:0/2149580570 >> 172.21.15.53:0/2149580570 conn(0x7f3cf4104370 msgr2=0x7f3cf41067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.087+0000 7f3cf9019700 1 -- 172.21.15.53:0/2149580570 shutdown_connections 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.087+0000 7f3cf9019700 1 -- 172.21.15.53:0/2149580570 wait complete. 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.087+0000 7f3cf9019700 1 Processor -- start 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.087+0000 7f3cf9019700 1 -- start start 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cf9019700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cf9019700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf410f310 con 0x7f3cf407f680 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cf2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cf2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38018/0 (socket says 172.21.15.53:38018) 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cf2d9d700 1 -- 172.21.15.53:0/3962007137 learned_addr learned my addr 172.21.15.53:0/3962007137 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cf2d9d700 1 -- 172.21.15.53:0/3962007137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf419f390 con 0x7f3cf407f680 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cf2d9d700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf419ee50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8004660 tx=0x7f3ce8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:39.290 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.088+0000 7f3cdbfff700 1 -- 172.21.15.53:0/3962007137 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce8003aa0 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.089+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf419f620 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.089+0000 7f3cdbfff700 1 -- 172.21.15.53:0/3962007137 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ce801b8f0 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.089+0000 7f3cdbfff700 1 -- 172.21.15.53:0/3962007137 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce8023900 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.089+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf41a2310 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.089+0000 7f3cdbfff700 1 -- 172.21.15.53:0/3962007137 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f3ce8023bc0 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.090+0000 7f3cdbfff700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3cdc039600 0x7f3cdc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.090+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf4064a80 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.090+0000 7f3cdbfff700 1 -- 172.21.15.53:0/3962007137 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3ce804eff0 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.090+0000 7f3cf259c700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3cdc039600 0x7f3cdc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.090+0000 7f3cf259c700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3cdc039600 0x7f3cdc03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4006fd0 tx=0x7f3ce4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.095+0000 7f3cdbfff700 1 -- 172.21.15.53:0/3962007137 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3ce801f030 con 0x7f3cf407f680 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.340+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f3cf419fb00 con 0x7f3cdc039600 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.587+0000 7f3cdbfff700 1 -- 172.21.15.53:0/3962007137 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f3cf419fb00 con 0x7f3cdc039600 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3cdc039600 msgr2=0x7f3cdc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3cdc039600 0x7f3cdc03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4006fd0 tx=0x7f3ce4006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 msgr2=0x7f3cf419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf419ee50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8004660 tx=0x7f3ce8011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 shutdown_connections 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3cdc039600 0x7f3cdc03bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 --2- 172.21.15.53:0/3962007137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cf407f680 0x7f3cf419ee50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 >> 172.21.15.53:0/3962007137 conn(0x7f3cf4104370 msgr2=0x7f3cf4104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 shutdown_connections 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:38.589+0000 7f3cf9019700 1 -- 172.21.15.53:0/3962007137 wait complete. 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi053.stdout:Fetching dashboard port number... 2024-04-28T12:41:39.843 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:39 smithi053 ceph-mon[29182]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:39.843 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:40.862 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:40 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2813601820' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 8443 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b229e2700 1 Processor -- start 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b229e2700 1 -- start start 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b229e2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b229e2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1097a0 con 0x7f8b1c108db0 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38030/0 (socket says 172.21.15.53:38030) 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b1bfff700 1 -- 172.21.15.53:0/2703418749 learned_addr learned my addr 172.21.15.53:0/2703418749 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.125+0000 7f8b1bfff700 1 -- 172.21.15.53:0/2703418749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1c109f60 con 0x7f8b1c108db0 2024-04-28T12:41:40.871 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.126+0000 7f8b1bfff700 1 --2- 172.21.15.53:0/2703418749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c1091d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0098a0 tx=0x7f8b0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad34381549cdb3ec server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.126+0000 7f8b1affd700 1 -- 172.21.15.53:0/2703418749 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c004160 con 0x7f8b1c108db0 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.126+0000 7f8b1affd700 1 -- 172.21.15.53:0/2703418749 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b0c004300 con 0x7f8b1c108db0 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.126+0000 7f8b1affd700 1 -- 172.21.15.53:0/2703418749 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c011a20 con 0x7f8b1c108db0 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.126+0000 7f8b229e2700 1 -- 172.21.15.53:0/2703418749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 msgr2=0x7f8b1c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.126+0000 7f8b229e2700 1 --2- 172.21.15.53:0/2703418749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c1091d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0098a0 tx=0x7f8b0c009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.127+0000 7f8b229e2700 1 -- 172.21.15.53:0/2703418749 shutdown_connections 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.127+0000 7f8b229e2700 1 --2- 172.21.15.53:0/2703418749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c1091d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.127+0000 7f8b229e2700 1 -- 172.21.15.53:0/2703418749 >> 172.21.15.53:0/2703418749 conn(0x7f8b1c104310 msgr2=0x7f8b1c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.127+0000 7f8b229e2700 1 -- 172.21.15.53:0/2703418749 shutdown_connections 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.127+0000 7f8b229e2700 1 -- 172.21.15.53:0/2703418749 wait complete. 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.127+0000 7f8b229e2700 1 Processor -- start 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.127+0000 7f8b229e2700 1 -- start start 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b229e2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c19cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:40.872 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b229e2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c10d310 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c19cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b1bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c19cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38034/0 (socket says 172.21.15.53:38034) 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b1bfff700 1 -- 172.21.15.53:0/2813601820 learned_addr learned my addr 172.21.15.53:0/2813601820 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b1bfff700 1 -- 172.21.15.53:0/2813601820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1c19d1c0 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b1bfff700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c19cc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c000c00 tx=0x7f8b0c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b197fa700 1 -- 172.21.15.53:0/2813601820 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c003aa0 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b197fa700 1 -- 172.21.15.53:0/2813601820 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b0c01b8f0 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1c19d450 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b197fa700 1 -- 172.21.15.53:0/2813601820 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c023890 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.128+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1c1a0140 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.129+0000 7f8b197fa700 1 -- 172.21.15.53:0/2813601820 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f8b0c02c3f0 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.129+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b1c064a80 con 0x7f8b1c108db0 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.129+0000 7f8b197fa700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8b04039660 0x7f8b0403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.130+0000 7f8b197fa700 1 -- 172.21.15.53:0/2813601820 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8b0c04df50 con 0x7f8b1c108db0 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.130+0000 7f8b1b7fe700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8b04039660 0x7f8b0403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.130+0000 7f8b1b7fe700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8b04039660 0x7f8b0403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8b10006fd0 tx=0x7f8b10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.135+0000 7f8b197fa700 1 -- 172.21.15.53:0/2813601820 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8b0c01f030 con 0x7f8b1c108db0 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.342+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f8b1c1a0730 con 0x7f8b1c108db0 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.343+0000 7f8b197fa700 1 -- 172.21.15.53:0/2813601820 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f8b0c0480b0 con 0x7f8b1c108db0 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8b04039660 msgr2=0x7f8b0403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8b04039660 0x7f8b0403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8b10006fd0 tx=0x7f8b10006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 msgr2=0x7f8b1c19cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c19cc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c000c00 tx=0x7f8b0c011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 shutdown_connections 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8b04039660 0x7f8b0403bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 --2- 172.21.15.53:0/2813601820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b1c108db0 0x7f8b1c19cc80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:40.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 >> 172.21.15.53:0/2813601820 conn(0x7f8b1c104310 msgr2=0x7f8b1c104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:40.875 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 shutdown_connections 2024-04-28T12:41:40.875 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:40.345+0000 7f8b229e2700 1 -- 172.21.15.53:0/2813601820 wait complete. 2024-04-28T12:41:40.878 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:41:40.878 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-04-28T12:41:40.885 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:41:40.885 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-04-28T12:41:40.885 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-04-28T12:41:40.885 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-28T12:41:40.887 INFO:teuthology.orchestra.run.smithi053.stdout:Ceph Dashboard is now available at: 2024-04-28T12:41:40.887 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:40.887 INFO:teuthology.orchestra.run.smithi053.stdout: URL: https://smithi053:8443/ 2024-04-28T12:41:40.887 INFO:teuthology.orchestra.run.smithi053.stdout: User: admin 2024-04-28T12:41:40.887 INFO:teuthology.orchestra.run.smithi053.stdout: Password: oujah6rjhe 2024-04-28T12:41:40.887 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:40.887 INFO:teuthology.orchestra.run.smithi053.stdout:Saving cluster configuration to /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config directory 2024-04-28T12:41:40.888 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling autotune for osd_memory_target 2024-04-28T12:41:41.810 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:41 smithi053 ceph-mon[29182]: mgrmap e13: x(active, since 6s) 2024-04-28T12:41:42.034 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.308+0000 7fb43d0c5700 1 Processor -- start 2024-04-28T12:41:42.034 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.308+0000 7fb43d0c5700 1 -- start start 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb43d0c5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb438109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb43d0c5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb438109760 con 0x7fb438108d70 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb436d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb438109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb436d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb438109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38044/0 (socket says 172.21.15.53:38044) 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb436d9d700 1 -- 172.21.15.53:0/2383983983 learned_addr learned my addr 172.21.15.53:0/2383983983 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb436d9d700 1 -- 172.21.15.53:0/2383983983 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb438109f70 con 0x7fb438108d70 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb436d9d700 1 --2- 172.21.15.53:0/2383983983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb438109190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4280098a0 tx=0x7fb428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a359e36e39fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.309+0000 7fb435d9b700 1 -- 172.21.15.53:0/2383983983 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb428004160 con 0x7fb438108d70 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb435d9b700 1 -- 172.21.15.53:0/2383983983 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb428004300 con 0x7fb438108d70 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb435d9b700 1 -- 172.21.15.53:0/2383983983 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb428011a20 con 0x7fb438108d70 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2383983983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 msgr2=0x7fb438109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb43d0c5700 1 --2- 172.21.15.53:0/2383983983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb438109190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4280098a0 tx=0x7fb428009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.035 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2383983983 shutdown_connections 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb43d0c5700 1 --2- 172.21.15.53:0/2383983983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb438109190 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2383983983 >> 172.21.15.53:0/2383983983 conn(0x7fb4381042d0 msgr2=0x7fb438106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2383983983 shutdown_connections 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.310+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2383983983 wait complete. 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.311+0000 7fb43d0c5700 1 Processor -- start 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.311+0000 7fb43d0c5700 1 -- start start 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb43d0c5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb43819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb43d0c5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb43810d320 con 0x7fb438108d70 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb436d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb43819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb436d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb43819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38060/0 (socket says 172.21.15.53:38060) 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb436d9d700 1 -- 172.21.15.53:0/2156157569 learned_addr learned my addr 172.21.15.53:0/2156157569 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb436d9d700 1 -- 172.21.15.53:0/2156157569 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43819d200 con 0x7fb438108d70 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb436d9d700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb43819ccc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4280119f0 tx=0x7fb428011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:42.036 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.312+0000 7fb42ffff700 1 -- 172.21.15.53:0/2156157569 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4280039e0 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.313+0000 7fb42ffff700 1 -- 172.21.15.53:0/2156157569 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb42801b8f0 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.313+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43819d490 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.313+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4381a0180 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.313+0000 7fb42ffff700 1 -- 172.21.15.53:0/2156157569 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4280237a0 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.313+0000 7fb42ffff700 1 -- 172.21.15.53:0/2156157569 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb42802c3f0 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.314+0000 7fb42ffff700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb4240396b0 0x7fb42403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.314+0000 7fb42ffff700 1 -- 172.21.15.53:0/2156157569 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb42801bd60 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.314+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb438064a80 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.314+0000 7fb43659c700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb4240396b0 0x7fb42403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.314+0000 7fb43659c700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb4240396b0 0x7fb42403bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb420006fd0 tx=0x7fb420006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.319+0000 7fb42ffff700 1 -- 172.21.15.53:0/2156157569 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb42801f030 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.528+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb4381a06e0 con 0x7fb438108d70 2024-04-28T12:41:42.037 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.529+0000 7fb42ffff700 1 -- 172.21.15.53:0/2156157569 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb42802b7a0 con 0x7fb438108d70 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb4240396b0 msgr2=0x7fb42403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb4240396b0 0x7fb42403bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb420006fd0 tx=0x7fb420006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 msgr2=0x7fb43819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb43819ccc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4280119f0 tx=0x7fb428011ad0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 shutdown_connections 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb4240396b0 0x7fb42403bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 --2- 172.21.15.53:0/2156157569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb438108d70 0x7fb43819ccc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 >> 172.21.15.53:0/2156157569 conn(0x7fb4381042d0 msgr2=0x7fb438104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 shutdown_connections 2024-04-28T12:41:42.038 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:41.532+0000 7fb43d0c5700 1 -- 172.21.15.53:0/2156157569 wait complete. 2024-04-28T12:41:43.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.413+0000 7f22c6ef9700 1 Processor -- start 2024-04-28T12:41:43.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.413+0000 7f22c6ef9700 1 -- start start 2024-04-28T12:41:43.291 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.413+0000 7f22c6ef9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c010bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.413+0000 7f22c6ef9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f22c007d6f0 con 0x7f22c0109800 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.413+0000 7f22c4c95700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c010bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.413+0000 7f22c4c95700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c010bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38068/0 (socket says 172.21.15.53:38068) 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.413+0000 7f22c4c95700 1 -- 172.21.15.53:0/905156953 learned_addr learned my addr 172.21.15.53:0/905156953 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.414+0000 7f22c4c95700 1 -- 172.21.15.53:0/905156953 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c010c130 con 0x7f22c0109800 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.414+0000 7f22c4c95700 1 --2- 172.21.15.53:0/905156953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c010bbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0098a0 tx=0x7f22bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb890c969745a41 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.414+0000 7f22b77fe700 1 -- 172.21.15.53:0/905156953 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc004160 con 0x7f22c0109800 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.414+0000 7f22b77fe700 1 -- 172.21.15.53:0/905156953 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22bc004300 con 0x7f22c0109800 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.414+0000 7f22b77fe700 1 -- 172.21.15.53:0/905156953 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc011a20 con 0x7f22c0109800 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 -- 172.21.15.53:0/905156953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 msgr2=0x7f22c010bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 --2- 172.21.15.53:0/905156953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c010bbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0098a0 tx=0x7f22bc009bb0 comp rx=0 tx=0).stop 2024-04-28T12:41:43.292 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 -- 172.21.15.53:0/905156953 shutdown_connections 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 --2- 172.21.15.53:0/905156953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c010bbf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 -- 172.21.15.53:0/905156953 >> 172.21.15.53:0/905156953 conn(0x7f22c01031f0 msgr2=0x7f22c0105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 -- 172.21.15.53:0/905156953 shutdown_connections 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 -- 172.21.15.53:0/905156953 wait complete. 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 Processor -- start 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 -- start start 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.415+0000 7f22c6ef9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f22c010d130 con 0x7f22c0109800 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22c4c95700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22c4c95700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c019cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38082/0 (socket says 172.21.15.53:38082) 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22c4c95700 1 -- 172.21.15.53:0/3317553083 learned_addr learned my addr 172.21.15.53:0/3317553083 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:43.293 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22c4c95700 1 -- 172.21.15.53:0/3317553083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c019d1a0 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22c4c95700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c019cc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0116e0 tx=0x7f22bc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22b5ffb700 1 -- 172.21.15.53:0/3317553083 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc003b20 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22b5ffb700 1 -- 172.21.15.53:0/3317553083 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22bc01b8f0 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22b5ffb700 1 -- 172.21.15.53:0/3317553083 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22bc0238c0 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c019d430 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.416+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c01a0120 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.417+0000 7f22b5ffb700 1 -- 172.21.15.53:0/3317553083 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f22bc01baa0 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.417+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c0064a80 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.417+0000 7f22b5ffb700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f22a8039650 0x7f22a803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.417+0000 7f22b5ffb700 1 -- 172.21.15.53:0/3317553083 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f22bc04e270 con 0x7f22c0109800 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.418+0000 7f22b7fff700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f22a8039650 0x7f22a803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.418+0000 7f22b7fff700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f22a8039650 0x7f22a803bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f22b0006fd0 tx=0x7f22b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.422+0000 7f22b5ffb700 1 -- 172.21.15.53:0/3317553083 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f22bc01f030 con 0x7f22c0109800 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.745+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f22c01a02f0 con 0x7f22c0109800 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.747+0000 7f22b5ffb700 1 -- 172.21.15.53:0/3317553083 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f22bc018350 con 0x7f22c0109800 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f22a8039650 msgr2=0x7f22a803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f22a8039650 0x7f22a803bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f22b0006fd0 tx=0x7f22b0006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 msgr2=0x7f22c019cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c019cc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0116e0 tx=0x7f22bc011aa0 comp rx=0 tx=0).stop 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 shutdown_connections 2024-04-28T12:41:43.295 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f22a8039650 0x7f22a803bb10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 --2- 172.21.15.53:0/3317553083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c0109800 0x7f22c019cc60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 >> 172.21.15.53:0/3317553083 conn(0x7f22c01031f0 msgr2=0x7f22c0103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 shutdown_connections 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2024-04-28T12:41:42.750+0000 7f22c6ef9700 1 -- 172.21.15.53:0/3317553083 wait complete. 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:Or, if you are only running a single cluster on this host: 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-28T12:41:43.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.297 INFO:teuthology.orchestra.run.smithi053.stdout: ceph telemetry on 2024-04-28T12:41:43.297 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.297 INFO:teuthology.orchestra.run.smithi053.stdout:For more information see: 2024-04-28T12:41:43.297 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.297 INFO:teuthology.orchestra.run.smithi053.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-28T12:41:43.297 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:43.297 INFO:teuthology.orchestra.run.smithi053.stdout:Bootstrap complete. 2024-04-28T12:41:43.332 INFO:tasks.cephadm:Fetching config... 2024-04-28T12:41:43.333 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:41:43.333 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-28T12:41:43.350 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-28T12:41:43.350 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:41:43.350 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-28T12:41:43.406 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-28T12:41:43.406 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:41:43.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-28T12:41:43.471 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-28T12:41:43.471 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:41:43.472 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-28T12:41:43.527 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-28T12:41:43.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDw3vQU3JplBaj/sY50eKDX6yb/S8VrzD+EWWaUhoW7C2UShf4tolhLe7Y9cEEFd+ieJxWmOKmjy21slNSve2MYitgRXKNG9MixQaRoScpF08HwNedCK5SL+9SCCD7dhlmd//o1Fb6rs1WluhSs3a2RmWfvp4zWR4LNLm15Dp2ViBcHgIYeKl7MmKcm2M7Kk4XzIW+b3Ukdt/4hcQEJGyXggR4VEYuU63WeoZRismYs2pE8m8mSMwQVpUYDkgC00Y4RizxJ/ajps+FE7/IyXkPoFTGuHGCXjDpifxgESoHTv2XSguHdr6tePN2jhkQUXExfvN0JAt5AKLK3X22hJIEqucpNSDyymqlsZG8QLg+DyNWMldB4Uw1VAqAMeralEc9XajPKY0RvI6S2OS98oaWcrKuBljzBbQSdfvRwsv5+3Wbgvua5bhLqxnoHA9WZ89kU4sbFsDdi4DJk172olpUP7uiWe5bPP0c5sCsTqFFdBOAoYoIFFa6sT0YUtOe78Y0= ceph-10525726-055c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T12:41:43.605 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDw3vQU3JplBaj/sY50eKDX6yb/S8VrzD+EWWaUhoW7C2UShf4tolhLe7Y9cEEFd+ieJxWmOKmjy21slNSve2MYitgRXKNG9MixQaRoScpF08HwNedCK5SL+9SCCD7dhlmd//o1Fb6rs1WluhSs3a2RmWfvp4zWR4LNLm15Dp2ViBcHgIYeKl7MmKcm2M7Kk4XzIW+b3Ukdt/4hcQEJGyXggR4VEYuU63WeoZRismYs2pE8m8mSMwQVpUYDkgC00Y4RizxJ/ajps+FE7/IyXkPoFTGuHGCXjDpifxgESoHTv2XSguHdr6tePN2jhkQUXExfvN0JAt5AKLK3X22hJIEqucpNSDyymqlsZG8QLg+DyNWMldB4Uw1VAqAMeralEc9XajPKY0RvI6S2OS98oaWcrKuBljzBbQSdfvRwsv5+3Wbgvua5bhLqxnoHA9WZ89kU4sbFsDdi4DJk172olpUP7uiWe5bPP0c5sCsTqFFdBOAoYoIFFa6sT0YUtOe78Y0= ceph-10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:41:43.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDw3vQU3JplBaj/sY50eKDX6yb/S8VrzD+EWWaUhoW7C2UShf4tolhLe7Y9cEEFd+ieJxWmOKmjy21slNSve2MYitgRXKNG9MixQaRoScpF08HwNedCK5SL+9SCCD7dhlmd//o1Fb6rs1WluhSs3a2RmWfvp4zWR4LNLm15Dp2ViBcHgIYeKl7MmKcm2M7Kk4XzIW+b3Ukdt/4hcQEJGyXggR4VEYuU63WeoZRismYs2pE8m8mSMwQVpUYDkgC00Y4RizxJ/ajps+FE7/IyXkPoFTGuHGCXjDpifxgESoHTv2XSguHdr6tePN2jhkQUXExfvN0JAt5AKLK3X22hJIEqucpNSDyymqlsZG8QLg+DyNWMldB4Uw1VAqAMeralEc9XajPKY0RvI6S2OS98oaWcrKuBljzBbQSdfvRwsv5+3Wbgvua5bhLqxnoHA9WZ89kU4sbFsDdi4DJk172olpUP7uiWe5bPP0c5sCsTqFFdBOAoYoIFFa6sT0YUtOe78Y0= ceph-10525726-055c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T12:41:43.656 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDw3vQU3JplBaj/sY50eKDX6yb/S8VrzD+EWWaUhoW7C2UShf4tolhLe7Y9cEEFd+ieJxWmOKmjy21slNSve2MYitgRXKNG9MixQaRoScpF08HwNedCK5SL+9SCCD7dhlmd//o1Fb6rs1WluhSs3a2RmWfvp4zWR4LNLm15Dp2ViBcHgIYeKl7MmKcm2M7Kk4XzIW+b3Ukdt/4hcQEJGyXggR4VEYuU63WeoZRismYs2pE8m8mSMwQVpUYDkgC00Y4RizxJ/ajps+FE7/IyXkPoFTGuHGCXjDpifxgESoHTv2XSguHdr6tePN2jhkQUXExfvN0JAt5AKLK3X22hJIEqucpNSDyymqlsZG8QLg+DyNWMldB4Uw1VAqAMeralEc9XajPKY0RvI6S2OS98oaWcrKuBljzBbQSdfvRwsv5+3Wbgvua5bhLqxnoHA9WZ89kU4sbFsDdi4DJk172olpUP7uiWe5bPP0c5sCsTqFFdBOAoYoIFFa6sT0YUtOe78Y0= ceph-10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:41:43.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDw3vQU3JplBaj/sY50eKDX6yb/S8VrzD+EWWaUhoW7C2UShf4tolhLe7Y9cEEFd+ieJxWmOKmjy21slNSve2MYitgRXKNG9MixQaRoScpF08HwNedCK5SL+9SCCD7dhlmd//o1Fb6rs1WluhSs3a2RmWfvp4zWR4LNLm15Dp2ViBcHgIYeKl7MmKcm2M7Kk4XzIW+b3Ukdt/4hcQEJGyXggR4VEYuU63WeoZRismYs2pE8m8mSMwQVpUYDkgC00Y4RizxJ/ajps+FE7/IyXkPoFTGuHGCXjDpifxgESoHTv2XSguHdr6tePN2jhkQUXExfvN0JAt5AKLK3X22hJIEqucpNSDyymqlsZG8QLg+DyNWMldB4Uw1VAqAMeralEc9XajPKY0RvI6S2OS98oaWcrKuBljzBbQSdfvRwsv5+3Wbgvua5bhLqxnoHA9WZ89kU4sbFsDdi4DJk172olpUP7uiWe5bPP0c5sCsTqFFdBOAoYoIFFa6sT0YUtOe78Y0= ceph-10525726-055c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T12:41:43.707 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDw3vQU3JplBaj/sY50eKDX6yb/S8VrzD+EWWaUhoW7C2UShf4tolhLe7Y9cEEFd+ieJxWmOKmjy21slNSve2MYitgRXKNG9MixQaRoScpF08HwNedCK5SL+9SCCD7dhlmd//o1Fb6rs1WluhSs3a2RmWfvp4zWR4LNLm15Dp2ViBcHgIYeKl7MmKcm2M7Kk4XzIW+b3Ukdt/4hcQEJGyXggR4VEYuU63WeoZRismYs2pE8m8mSMwQVpUYDkgC00Y4RizxJ/ajps+FE7/IyXkPoFTGuHGCXjDpifxgESoHTv2XSguHdr6tePN2jhkQUXExfvN0JAt5AKLK3X22hJIEqucpNSDyymqlsZG8QLg+DyNWMldB4Uw1VAqAMeralEc9XajPKY0RvI6S2OS98oaWcrKuBljzBbQSdfvRwsv5+3Wbgvua5bhLqxnoHA9WZ89kU4sbFsDdi4DJk172olpUP7uiWe5bPP0c5sCsTqFFdBOAoYoIFFa6sT0YUtOe78Y0= ceph-10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:41:43.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-28T12:41:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3317553083' entity='client.admin' 2024-04-28T12:41:43.969 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:46.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:46.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:46.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:41:46.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:46.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:46.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:46.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.699+0000 7ffa6bf30700 1 -- 172.21.15.53:0/2510968827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 msgr2=0x7ffa641021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:46.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.699+0000 7ffa6bf30700 1 --2- 172.21.15.53:0/2510968827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa641021f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009910 tx=0x7ffa60009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:46.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.699+0000 7ffa6bf30700 1 -- 172.21.15.53:0/2510968827 shutdown_connections 2024-04-28T12:41:46.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.699+0000 7ffa6bf30700 1 --2- 172.21.15.53:0/2510968827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa641021f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.699+0000 7ffa6bf30700 1 -- 172.21.15.53:0/2510968827 >> 172.21.15.53:0/2510968827 conn(0x7ffa640fd330 msgr2=0x7ffa640ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:46.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.699+0000 7ffa6bf30700 1 -- 172.21.15.53:0/2510968827 shutdown_connections 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.699+0000 7ffa6bf30700 1 -- 172.21.15.53:0/2510968827 wait complete. 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa6bf30700 1 Processor -- start 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa6bf30700 1 -- start start 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa6bf30700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa640ff230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa6bf30700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffa64107d10 con 0x7ffa64101dd0 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa69ccc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa640ff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa69ccc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa640ff230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37516/0 (socket says 172.21.15.53:37516) 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa69ccc700 1 -- 172.21.15.53:0/4227418494 learned_addr learned my addr 172.21.15.53:0/4227418494 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.700+0000 7ffa69ccc700 1 -- 172.21.15.53:0/4227418494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa640ff770 con 0x7ffa64101dd0 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.701+0000 7ffa69ccc700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa640ff230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005800 tx=0x7ffa60009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.701+0000 7ffa5affd700 1 -- 172.21.15.53:0/4227418494 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa60027070 con 0x7ffa64101dd0 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.701+0000 7ffa5affd700 1 -- 172.21.15.53:0/4227418494 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa6000be10 con 0x7ffa64101dd0 2024-04-28T12:41:46.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.701+0000 7ffa5affd700 1 -- 172.21.15.53:0/4227418494 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa6000f460 con 0x7ffa64101dd0 2024-04-28T12:41:46.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.701+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa640fd960 con 0x7ffa64101dd0 2024-04-28T12:41:46.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.701+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa640fdea0 con 0x7ffa64101dd0 2024-04-28T12:41:46.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.702+0000 7ffa5affd700 1 -- 172.21.15.53:0/4227418494 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ffa6002a070 con 0x7ffa64101dd0 2024-04-28T12:41:46.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.702+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa64108490 con 0x7ffa64101dd0 2024-04-28T12:41:46.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.702+0000 7ffa5affd700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffa500396b0 0x7ffa5003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:46.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.702+0000 7ffa5affd700 1 -- 172.21.15.53:0/4227418494 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffa6001e030 con 0x7ffa64101dd0 2024-04-28T12:41:46.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.702+0000 7ffa694cb700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffa500396b0 0x7ffa5003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:46.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.702+0000 7ffa694cb700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffa500396b0 0x7ffa5003bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffa54006fd0 tx=0x7ffa54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:46.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.707+0000 7ffa5affd700 1 -- 172.21.15.53:0/4227418494 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffa60013070 con 0x7ffa64101dd0 2024-04-28T12:41:46.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.913+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7ffa64191a30 con 0x7ffa64101dd0 2024-04-28T12:41:46.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.916+0000 7ffa5affd700 1 -- 172.21.15.53:0/4227418494 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7ffa60031020 con 0x7ffa64101dd0 2024-04-28T12:41:46.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffa500396b0 msgr2=0x7ffa5003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:46.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffa500396b0 0x7ffa5003bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffa54006fd0 tx=0x7ffa54006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:46.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 msgr2=0x7ffa640ff230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:46.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa640ff230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005800 tx=0x7ffa60009f60 comp rx=0 tx=0).stop 2024-04-28T12:41:46.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 shutdown_connections 2024-04-28T12:41:46.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffa500396b0 0x7ffa5003bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 --2- 172.21.15.53:0/4227418494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64101dd0 0x7ffa640ff230 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 >> 172.21.15.53:0/4227418494 conn(0x7ffa640fd330 msgr2=0x7ffa6406b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:46.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 shutdown_connections 2024-04-28T12:41:46.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:46.918+0000 7ffa6bf30700 1 -- 172.21.15.53:0/4227418494 wait complete. 2024-04-28T12:41:47.767 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-28T12:41:47.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-28T12:41:48.014 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:48.220 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4227418494' entity='client.admin' 2024-04-28T12:41:48.220 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:48.221 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:49.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.436+0000 7f4b83e79700 1 -- 172.21.15.53:0/1446332980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 msgr2=0x7f4b7c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:49.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.436+0000 7f4b83e79700 1 --2- 172.21.15.53:0/1446332980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c101b40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009910 tx=0x7f4b78009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:49.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.437+0000 7f4b83e79700 1 -- 172.21.15.53:0/1446332980 shutdown_connections 2024-04-28T12:41:49.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.437+0000 7f4b83e79700 1 --2- 172.21.15.53:0/1446332980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c101b40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:49.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.437+0000 7f4b83e79700 1 -- 172.21.15.53:0/1446332980 >> 172.21.15.53:0/1446332980 conn(0x7f4b7c0fd360 msgr2=0x7f4b7c0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:49.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.437+0000 7f4b83e79700 1 -- 172.21.15.53:0/1446332980 shutdown_connections 2024-04-28T12:41:49.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.437+0000 7f4b83e79700 1 -- 172.21.15.53:0/1446332980 wait complete. 2024-04-28T12:41:49.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.437+0000 7f4b83e79700 1 Processor -- start 2024-04-28T12:41:49.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.437+0000 7f4b83e79700 1 -- start start 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b83e79700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b83e79700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c10afd0 con 0x7f4b7c101720 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b81c15700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b81c15700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c19a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37534/0 (socket says 172.21.15.53:37534) 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b81c15700 1 -- 172.21.15.53:0/3861496289 learned_addr learned my addr 172.21.15.53:0/3861496289 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b81c15700 1 -- 172.21.15.53:0/3861496289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c19ac90 con 0x7f4b7c101720 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b81c15700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c19a750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b78005b40 tx=0x7f4b78004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:49.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b72ffd700 1 -- 172.21.15.53:0/3861496289 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b78027070 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7c19af20 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b72ffd700 1 -- 172.21.15.53:0/3861496289 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b7800be10 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b72ffd700 1 -- 172.21.15.53:0/3861496289 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b78022d80 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.438+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7c19dc10 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.439+0000 7f4b72ffd700 1 -- 172.21.15.53:0/3861496289 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4b7802a070 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.439+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b7c10b750 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.439+0000 7f4b72ffd700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4b68039650 0x7f4b6803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.439+0000 7f4b72ffd700 1 -- 172.21.15.53:0/3861496289 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4b7801e030 con 0x7f4b7c101720 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.440+0000 7f4b81414700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4b68039650 0x7f4b6803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.440+0000 7f4b81414700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4b68039650 0x7f4b6803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c006fd0 tx=0x7f4b6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:49.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.445+0000 7f4b72ffd700 1 -- 172.21.15.53:0/3861496289 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4b78013070 con 0x7f4b7c101720 2024-04-28T12:41:49.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.657+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f4b7c02e5a0 con 0x7f4b68039650 2024-04-28T12:41:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.660+0000 7f4b72ffd700 1 -- 172.21.15.53:0/3861496289 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4b7c02e5a0 con 0x7f4b68039650 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4b68039650 msgr2=0x7f4b6803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4b68039650 0x7f4b6803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c006fd0 tx=0x7f4b6c006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 msgr2=0x7f4b7c19a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c19a750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b78005b40 tx=0x7f4b78004f80 comp rx=0 tx=0).stop 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 shutdown_connections 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4b68039650 0x7f4b6803bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 --2- 172.21.15.53:0/3861496289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b7c101720 0x7f4b7c19a750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.662+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 >> 172.21.15.53:0/3861496289 conn(0x7f4b7c0fd360 msgr2=0x7f4b7c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.663+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 shutdown_connections 2024-04-28T12:41:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:49.663+0000 7f4b83e79700 1 -- 172.21.15.53:0/3861496289 wait complete. 2024-04-28T12:41:50.440 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi077 2024-04-28T12:41:50.440 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:41:50.440 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.conf 2024-04-28T12:41:50.459 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:41:50.459 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:50.516 INFO:tasks.cephadm:Adding host smithi077 to orchestrator... 2024-04-28T12:41:50.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch host add smithi077 2024-04-28T12:41:50.766 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: Updating smithi053:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:50.879 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:50.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:51.889 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:51 smithi053 ceph-mon[29182]: Updating smithi053:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:41:52.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.292+0000 7f00e40a1700 1 -- 172.21.15.53:0/1607068339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 msgr2=0x7f00dc102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:52.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.292+0000 7f00e40a1700 1 --2- 172.21.15.53:0/1607068339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc102370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009910 tx=0x7f00cc009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:52.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.293+0000 7f00e40a1700 1 -- 172.21.15.53:0/1607068339 shutdown_connections 2024-04-28T12:41:52.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.293+0000 7f00e40a1700 1 --2- 172.21.15.53:0/1607068339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc102370 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:52.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.293+0000 7f00e40a1700 1 -- 172.21.15.53:0/1607068339 >> 172.21.15.53:0/1607068339 conn(0x7f00dc0fd840 msgr2=0x7f00dc0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:52.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.293+0000 7f00e40a1700 1 -- 172.21.15.53:0/1607068339 shutdown_connections 2024-04-28T12:41:52.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.293+0000 7f00e40a1700 1 -- 172.21.15.53:0/1607068339 wait complete. 2024-04-28T12:41:52.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.293+0000 7f00e40a1700 1 Processor -- start 2024-04-28T12:41:52.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.293+0000 7f00e40a1700 1 -- start start 2024-04-28T12:41:52.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e40a1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:52.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e40a1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc10c210 con 0x7f00dc101f90 2024-04-28T12:41:52.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e1e3d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:52.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e1e3d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37552/0 (socket says 172.21.15.53:37552) 2024-04-28T12:41:52.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e1e3d700 1 -- 172.21.15.53:0/2794961289 learned_addr learned my addr 172.21.15.53:0/2794961289 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:52.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e1e3d700 1 -- 172.21.15.53:0/2794961289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc105920 con 0x7f00dc101f90 2024-04-28T12:41:52.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e1e3d700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc103a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0056d0 tx=0x7f00cc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:52.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00d2ffd700 1 -- 172.21.15.53:0/2794961289 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc026070 con 0x7f00dc101f90 2024-04-28T12:41:52.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00d2ffd700 1 -- 172.21.15.53:0/2794961289 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00cc00bc00 con 0x7f00dc101f90 2024-04-28T12:41:52.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00dc103fd0 con 0x7f00dc101f90 2024-04-28T12:41:52.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00d2ffd700 1 -- 172.21.15.53:0/2794961289 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00cc003920 con 0x7f00dc101f90 2024-04-28T12:41:52.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.294+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00dc104490 con 0x7f00dc101f90 2024-04-28T12:41:52.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.295+0000 7f00d2ffd700 1 -- 172.21.15.53:0/2794961289 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f00cc029070 con 0x7f00dc101f90 2024-04-28T12:41:52.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.295+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00dc103470 con 0x7f00dc101f90 2024-04-28T12:41:52.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.295+0000 7f00d2ffd700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f00c80396b0 0x7f00c803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:52.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.295+0000 7f00d2ffd700 1 -- 172.21.15.53:0/2794961289 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f00cc01e070 con 0x7f00dc101f90 2024-04-28T12:41:52.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.296+0000 7f00e163c700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f00c80396b0 0x7f00c803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:52.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.296+0000 7f00e163c700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f00c80396b0 0x7f00c803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f00d8006fd0 tx=0x7f00d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:52.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.300+0000 7f00d2ffd700 1 -- 172.21.15.53:0/2794961289 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f00cc013070 con 0x7f00dc101f90 2024-04-28T12:41:52.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:52.513+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}) -- 0x7f00dc104f60 con 0x7f00c80396b0 2024-04-28T12:41:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:53 smithi053 ceph-mon[29182]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:53 smithi053 ceph-mon[29182]: Deploying cephadm binary to smithi077 2024-04-28T12:41:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:55 smithi053 ceph-mon[29182]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:56.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.793+0000 7f00d2ffd700 1 -- 172.21.15.53:0/2794961289 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f00dc104f60 con 0x7f00c80396b0 2024-04-28T12:41:56.789 INFO:teuthology.orchestra.run.smithi053.stdout:Added host 'smithi077' with addr '172.21.15.77' 2024-04-28T12:41:56.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f00c80396b0 msgr2=0x7f00c803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:56.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f00c80396b0 0x7f00c803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f00d8006fd0 tx=0x7f00d8006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 msgr2=0x7f00dc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc103a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0056d0 tx=0x7f00cc0057b0 comp rx=0 tx=0).stop 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 shutdown_connections 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f00c80396b0 0x7f00c803bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 --2- 172.21.15.53:0/2794961289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00dc101f90 0x7f00dc103a30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 >> 172.21.15.53:0/2794961289 conn(0x7f00dc0fd840 msgr2=0x7f00dc06b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 shutdown_connections 2024-04-28T12:41:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:56.796+0000 7f00e40a1700 1 -- 172.21.15.53:0/2794961289 wait complete. 2024-04-28T12:41:57.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-28T12:41:57.663 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:58.118 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:57 smithi053 ceph-mon[29182]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:58.118 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:58.118 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:57 smithi053 ceph-mon[29182]: Added host smithi077 2024-04-28T12:41:58.119 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:58.119 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:59.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:58 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:41:59.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.208+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1495336541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 msgr2=0x7fbc34104470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:59.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.208+0000 7fbc3ad6c700 1 --2- 172.21.15.53:0/1495336541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc34104470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009910 tx=0x7fbc28009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:59.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.208+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1495336541 shutdown_connections 2024-04-28T12:41:59.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.208+0000 7fbc3ad6c700 1 --2- 172.21.15.53:0/1495336541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc34104470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:59.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.208+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1495336541 >> 172.21.15.53:0/1495336541 conn(0x7fbc340fd880 msgr2=0x7fbc340ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.208+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1495336541 shutdown_connections 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.208+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1495336541 wait complete. 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc3ad6c700 1 Processor -- start 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc3ad6c700 1 -- start start 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc3ad6c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc3419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc3ad6c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3410d3c0 con 0x7fbc34104090 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc38b08700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc3419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc38b08700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc3419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48354/0 (socket says 172.21.15.53:48354) 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc38b08700 1 -- 172.21.15.53:0/1257342113 learned_addr learned my addr 172.21.15.53:0/1257342113 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.209+0000 7fbc38b08700 1 -- 172.21.15.53:0/1257342113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3419eb80 con 0x7fbc34104090 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.210+0000 7fbc38b08700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc3419e640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbc280056d0 tx=0x7fbc280057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.210+0000 7fbc31ffb700 1 -- 172.21.15.53:0/1257342113 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc28026070 con 0x7fbc34104090 2024-04-28T12:41:59.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.210+0000 7fbc31ffb700 1 -- 172.21.15.53:0/1257342113 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc28004370 con 0x7fbc34104090 2024-04-28T12:41:59.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.210+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc3419ee10 con 0x7fbc34104090 2024-04-28T12:41:59.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.210+0000 7fbc31ffb700 1 -- 172.21.15.53:0/1257342113 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc280039b0 con 0x7fbc34104090 2024-04-28T12:41:59.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.210+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc34198710 con 0x7fbc34104090 2024-04-28T12:41:59.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.211+0000 7fbc31ffb700 1 -- 172.21.15.53:0/1257342113 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbc28029070 con 0x7fbc34104090 2024-04-28T12:41:59.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.211+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc3406bb10 con 0x7fbc34104090 2024-04-28T12:41:59.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.211+0000 7fbc31ffb700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbc240396b0 0x7fbc2403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:59.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.211+0000 7fbc31ffb700 1 -- 172.21.15.53:0/1257342113 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbc2801e070 con 0x7fbc34104090 2024-04-28T12:41:59.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.211+0000 7fbc33fff700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbc240396b0 0x7fbc2403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:59.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.212+0000 7fbc33fff700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbc240396b0 0x7fbc2403bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006fd0 tx=0x7fbc20006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:59.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.216+0000 7fbc31ffb700 1 -- 172.21.15.53:0/1257342113 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbc28013070 con 0x7fbc34104090 2024-04-28T12:41:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.432+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc34064a80 con 0x7fbc240396b0 2024-04-28T12:41:59.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.433+0000 7fbc31ffb700 1 -- 172.21.15.53:0/1257342113 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fbc34064a80 con 0x7fbc240396b0 2024-04-28T12:41:59.429 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:41:59.429 INFO:teuthology.orchestra.run.smithi053.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}] 2024-04-28T12:41:59.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbc240396b0 msgr2=0x7fbc2403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:59.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbc240396b0 0x7fbc2403bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006fd0 tx=0x7fbc20006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:59.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 msgr2=0x7fbc3419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc3419e640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbc280056d0 tx=0x7fbc280057b0 comp rx=0 tx=0).stop 2024-04-28T12:41:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 shutdown_connections 2024-04-28T12:41:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbc240396b0 0x7fbc2403bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 --2- 172.21.15.53:0/1257342113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc34104090 0x7fbc3419e640 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 >> 172.21.15.53:0/1257342113 conn(0x7fbc340fd880 msgr2=0x7fbc34106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.435+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 shutdown_connections 2024-04-28T12:41:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:41:59.436+0000 7fbc3ad6c700 1 -- 172.21.15.53:0/1257342113 wait complete. 2024-04-28T12:41:59.994 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:59 smithi053 ceph-mon[29182]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:59.995 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:41:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:00.025 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-04-28T12:42:00.026 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:42:00.026 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-04-28T12:42:00.045 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:42:00.045 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:00.102 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-04-28T12:42:00.102 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch host add smithi191 2024-04-28T12:42:00.349 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:42:01.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:00 smithi053 ceph-mon[29182]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:42:01.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 -- 172.21.15.53:0/3776482944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 msgr2=0x7fe8fc103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:01.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 --2- 172.21.15.53:0/3776482944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc103470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009910 tx=0x7fe8f8009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:01.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 -- 172.21.15.53:0/3776482944 shutdown_connections 2024-04-28T12:42:01.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 --2- 172.21.15.53:0/3776482944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc103470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:01.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 -- 172.21.15.53:0/3776482944 >> 172.21.15.53:0/3776482944 conn(0x7fe8fc0fe940 msgr2=0x7fe8fc100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:01.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 -- 172.21.15.53:0/3776482944 shutdown_connections 2024-04-28T12:42:01.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 -- 172.21.15.53:0/3776482944 wait complete. 2024-04-28T12:42:01.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 Processor -- start 2024-04-28T12:42:01.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.823+0000 7fe9034f9700 1 -- start start 2024-04-28T12:42:01.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe9034f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:01.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe9034f9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc10d310 con 0x7fe8fc103090 2024-04-28T12:42:01.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe901295700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:01.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe901295700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48368/0 (socket says 172.21.15.53:48368) 2024-04-28T12:42:01.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe901295700 1 -- 172.21.15.53:0/759154982 learned_addr learned my addr 172.21.15.53:0/759154982 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:42:01.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe901295700 1 -- 172.21.15.53:0/759154982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8fc077610 con 0x7fe8fc103090 2024-04-28T12:42:01.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe901295700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc075720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8006010 tx=0x7fe8f8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:01.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe8f27fc700 1 -- 172.21.15.53:0/759154982 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8f8026070 con 0x7fe8fc103090 2024-04-28T12:42:01.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.824+0000 7fe8f27fc700 1 -- 172.21.15.53:0/759154982 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8f800bc00 con 0x7fe8fc103090 2024-04-28T12:42:01.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.825+0000 7fe8f27fc700 1 -- 172.21.15.53:0/759154982 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8f8003910 con 0x7fe8fc103090 2024-04-28T12:42:01.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.825+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8fc075cc0 con 0x7fe8fc103090 2024-04-28T12:42:01.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.825+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8fc076180 con 0x7fe8fc103090 2024-04-28T12:42:01.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.826+0000 7fe8f27fc700 1 -- 172.21.15.53:0/759154982 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe8f8029070 con 0x7fe8fc103090 2024-04-28T12:42:01.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.826+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8fc104570 con 0x7fe8fc103090 2024-04-28T12:42:01.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.826+0000 7fe8f27fc700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe8e8039650 0x7fe8e803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:01.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.826+0000 7fe8f27fc700 1 -- 172.21.15.53:0/759154982 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe8f801e070 con 0x7fe8fc103090 2024-04-28T12:42:01.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.826+0000 7fe900a94700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe8e8039650 0x7fe8e803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:01.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.826+0000 7fe900a94700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe8e8039650 0x7fe8e803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec006fd0 tx=0x7fe8ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:01.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:01.831+0000 7fe8f27fc700 1 -- 172.21.15.53:0/759154982 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe8f8013070 con 0x7fe8fc103090 2024-04-28T12:42:02.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:02.045+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}) -- 0x7fe8fc076cc0 con 0x7fe8e8039650 2024-04-28T12:42:02.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:01 smithi053 ceph-mon[29182]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:03.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:03 smithi053 ceph-mon[29182]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:03 smithi053 ceph-mon[29182]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-28T12:42:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:03 smithi053 ceph-mon[29182]: Deploying cephadm binary to smithi191 2024-04-28T12:42:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:03 smithi053 ceph-mon[29182]: Updating smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:03 smithi053 ceph-mon[29182]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:05 smithi053 ceph-mon[29182]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:05 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:05 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:05 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:06.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.287+0000 7fe8f27fc700 1 -- 172.21.15.53:0/759154982 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe8fc076cc0 con 0x7fe8e8039650 2024-04-28T12:42:06.285 INFO:teuthology.orchestra.run.smithi053.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-04-28T12:42:06.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe8e8039650 msgr2=0x7fe8e803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:06.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe8e8039650 0x7fe8e803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec006fd0 tx=0x7fe8ec006e40 comp rx=0 tx=0).stop 2024-04-28T12:42:06.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 msgr2=0x7fe8fc075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:06.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc075720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8006010 tx=0x7fe8f8005780 comp rx=0 tx=0).stop 2024-04-28T12:42:06.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 shutdown_connections 2024-04-28T12:42:06.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe8e8039650 0x7fe8e803bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:06.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 --2- 172.21.15.53:0/759154982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc103090 0x7fe8fc075720 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:06.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 >> 172.21.15.53:0/759154982 conn(0x7fe8fc0fe940 msgr2=0x7fe8fc104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:06.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 shutdown_connections 2024-04-28T12:42:06.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:06.290+0000 7fe9034f9700 1 -- 172.21.15.53:0/759154982 wait complete. 2024-04-28T12:42:06.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:06 smithi053 ceph-mon[29182]: Updating smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:42:06.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:06.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:06.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:06.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-28T12:42:07.142 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:42:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:07 smithi053 ceph-mon[29182]: Added host smithi191 2024-04-28T12:42:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:07 smithi053 ceph-mon[29182]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:08.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.708+0000 7fbe55082700 1 -- 172.21.15.53:0/2366980088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 msgr2=0x7fbe501023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.708+0000 7fbe55082700 1 --2- 172.21.15.53:0/2366980088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe501023b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009910 tx=0x7fbe40009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.708+0000 7fbe55082700 1 -- 172.21.15.53:0/2366980088 shutdown_connections 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.708+0000 7fbe55082700 1 --2- 172.21.15.53:0/2366980088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe501023b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.708+0000 7fbe55082700 1 -- 172.21.15.53:0/2366980088 >> 172.21.15.53:0/2366980088 conn(0x7fbe50077880 msgr2=0x7fbe50077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.708+0000 7fbe55082700 1 -- 172.21.15.53:0/2366980088 shutdown_connections 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.708+0000 7fbe55082700 1 -- 172.21.15.53:0/2366980088 wait complete. 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe55082700 1 Processor -- start 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe55082700 1 -- start start 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe55082700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe5019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe55082700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5010c200 con 0x7fbe50101fd0 2024-04-28T12:42:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe4ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe5019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:08.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe4ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe5019e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51230/0 (socket says 172.21.15.53:51230) 2024-04-28T12:42:08.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe4ed9d700 1 -- 172.21.15.53:0/3770234261 learned_addr learned my addr 172.21.15.53:0/3770234261 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:42:08.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.709+0000 7fbe4ed9d700 1 -- 172.21.15.53:0/3770234261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5019eb00 con 0x7fbe50101fd0 2024-04-28T12:42:08.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.710+0000 7fbe4ed9d700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe5019e5c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe400056d0 tx=0x7fbe400057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:08.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.710+0000 7fbe4cd99700 1 -- 172.21.15.53:0/3770234261 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe40026070 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.710+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5019edf0 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.710+0000 7fbe4cd99700 1 -- 172.21.15.53:0/3770234261 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe4000bba0 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.710+0000 7fbe4cd99700 1 -- 172.21.15.53:0/3770234261 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe40003920 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.710+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe50198860 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.711+0000 7fbe4cd99700 1 -- 172.21.15.53:0/3770234261 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbe40029070 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.711+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe50103460 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.711+0000 7fbe4cd99700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbe3c039650 0x7fbe3c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.711+0000 7fbe4cd99700 1 -- 172.21.15.53:0/3770234261 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbe4001e070 con 0x7fbe50101fd0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.711+0000 7fbe47fff700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbe3c039650 0x7fbe3c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:08.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.711+0000 7fbe47fff700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbe3c039650 0x7fbe3c03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbe38006fd0 tx=0x7fbe38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:08.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.716+0000 7fbe4cd99700 1 -- 172.21.15.53:0/3770234261 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe40013070 con 0x7fbe50101fd0 2024-04-28T12:42:08.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.929+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe50199bc0 con 0x7fbe3c039650 2024-04-28T12:42:08.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.930+0000 7fbe4cd99700 1 -- 172.21.15.53:0/3770234261 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fbe50199bc0 con 0x7fbe3c039650 2024-04-28T12:42:08.927 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:42:08.927 INFO:teuthology.orchestra.run.smithi053.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbe3c039650 msgr2=0x7fbe3c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbe3c039650 0x7fbe3c03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbe38006fd0 tx=0x7fbe38006e40 comp rx=0 tx=0).stop 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 msgr2=0x7fbe5019e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe5019e5c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe400056d0 tx=0x7fbe400057b0 comp rx=0 tx=0).stop 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 shutdown_connections 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbe3c039650 0x7fbe3c03bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 --2- 172.21.15.53:0/3770234261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe50101fd0 0x7fbe5019e5c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 >> 172.21.15.53:0/3770234261 conn(0x7fbe50077880 msgr2=0x7fbe5006a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 shutdown_connections 2024-04-28T12:42:08.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:08.932+0000 7fbe55082700 1 -- 172.21.15.53:0/3770234261 wait complete. 2024-04-28T12:42:09.606 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-28T12:42:09.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-28T12:42:09.851 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:42:10.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:09 smithi053 ceph-mon[29182]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:10.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:10.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:09 smithi053 ceph-mon[29182]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:42:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.603+0000 7efcd2d8d700 1 -- 172.21.15.53:0/2752605581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 msgr2=0x7efccc1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.603+0000 7efcd2d8d700 1 --2- 172.21.15.53:0/2752605581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc1023b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009910 tx=0x7efcbc009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.603+0000 7efcd2d8d700 1 -- 172.21.15.53:0/2752605581 shutdown_connections 2024-04-28T12:42:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.603+0000 7efcd2d8d700 1 --2- 172.21.15.53:0/2752605581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc1023b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.603+0000 7efcd2d8d700 1 -- 172.21.15.53:0/2752605581 >> 172.21.15.53:0/2752605581 conn(0x7efccc077880 msgr2=0x7efccc077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.603+0000 7efcd2d8d700 1 -- 172.21.15.53:0/2752605581 shutdown_connections 2024-04-28T12:42:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.603+0000 7efcd2d8d700 1 -- 172.21.15.53:0/2752605581 wait complete. 2024-04-28T12:42:11.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.604+0000 7efcd2d8d700 1 Processor -- start 2024-04-28T12:42:11.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.604+0000 7efcd2d8d700 1 -- start start 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.604+0000 7efcd2d8d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.604+0000 7efcd2d8d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efccc10c200 con 0x7efccc101fd0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.604+0000 7efcd0b29700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.604+0000 7efcd0b29700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc19e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51248/0 (socket says 172.21.15.53:51248) 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.604+0000 7efcd0b29700 1 -- 172.21.15.53:0/706258250 learned_addr learned my addr 172.21.15.53:0/706258250 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.605+0000 7efcd0b29700 1 -- 172.21.15.53:0/706258250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc19eb00 con 0x7efccc101fd0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.605+0000 7efcd0b29700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc19e5c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0056d0 tx=0x7efcbc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.605+0000 7efcc9ffb700 1 -- 172.21.15.53:0/706258250 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcbc026070 con 0x7efccc101fd0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.605+0000 7efcc9ffb700 1 -- 172.21.15.53:0/706258250 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcbc00bba0 con 0x7efccc101fd0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.605+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efccc19edf0 con 0x7efccc101fd0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.605+0000 7efcc9ffb700 1 -- 172.21.15.53:0/706258250 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcbc003920 con 0x7efccc101fd0 2024-04-28T12:42:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.605+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efccc198860 con 0x7efccc101fd0 2024-04-28T12:42:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.606+0000 7efcc9ffb700 1 -- 172.21.15.53:0/706258250 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efcbc029070 con 0x7efccc101fd0 2024-04-28T12:42:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.606+0000 7efcc9ffb700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7efcb40396b0 0x7efcb403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:11.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.606+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efccc103460 con 0x7efccc101fd0 2024-04-28T12:42:11.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.606+0000 7efcc9ffb700 1 -- 172.21.15.53:0/706258250 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efcbc01e070 con 0x7efccc101fd0 2024-04-28T12:42:11.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.606+0000 7efccbfff700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7efcb40396b0 0x7efcb403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:11.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.607+0000 7efccbfff700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7efcb40396b0 0x7efcb403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcc0006fd0 tx=0x7efcc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:11.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.611+0000 7efcc9ffb700 1 -- 172.21.15.53:0/706258250 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efcbc013070 con 0x7efccc101fd0 2024-04-28T12:42:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.822+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7efccc0008d0 con 0x7efccc101fd0 2024-04-28T12:42:11.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.886+0000 7efcc9ffb700 1 -- 172.21.15.53:0/706258250 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7efcbc030030 con 0x7efccc101fd0 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.888+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7efcb40396b0 msgr2=0x7efcb403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.888+0000 7efcd2d8d700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7efcb40396b0 0x7efcb403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcc0006fd0 tx=0x7efcc0006e40 comp rx=0 tx=0).stop 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.888+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 msgr2=0x7efccc19e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.888+0000 7efcd2d8d700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc19e5c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0056d0 tx=0x7efcbc0057b0 comp rx=0 tx=0).stop 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.888+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 shutdown_connections 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.888+0000 7efcd2d8d700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7efcb40396b0 0x7efcb403bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.889+0000 7efcd2d8d700 1 --2- 172.21.15.53:0/706258250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efccc101fd0 0x7efccc19e5c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:11.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.889+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 >> 172.21.15.53:0/706258250 conn(0x7efccc077880 msgr2=0x7efccc06a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:11.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.889+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 shutdown_connections 2024-04-28T12:42:11.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:11.889+0000 7efcd2d8d700 1 -- 172.21.15.53:0/706258250 wait complete. 2024-04-28T12:42:11.886 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-04-28T12:42:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:11 smithi053 ceph-mon[29182]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:11 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/706258250' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-28T12:42:12.535 INFO:tasks.cephadm:Adding mon.a on smithi053 2024-04-28T12:42:12.535 INFO:tasks.cephadm:Adding mon.b on smithi077 2024-04-28T12:42:12.535 INFO:tasks.cephadm:Adding mon.c on smithi191 2024-04-28T12:42:12.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi053:172.21.15.53=a;smithi077:172.21.15.77=b;smithi191:172.21.15.191=c' 2024-04-28T12:42:12.774 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:13.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/706258250' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-28T12:42:13.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:13.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:13.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:13.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:13.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:13.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:13.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:13.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:13.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:12 smithi053 ceph-mon[29182]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-28T12:42:14.318 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 -- 172.21.15.191:0/2069406615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 msgr2=0x7fdb98109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 --2- 172.21.15.191:0/2069406615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb98109220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009910 tx=0x7fdb88009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 -- 172.21.15.191:0/2069406615 shutdown_connections 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 --2- 172.21.15.191:0/2069406615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb98109220 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 -- 172.21.15.191:0/2069406615 >> 172.21.15.191:0/2069406615 conn(0x7fdb980792b0 msgr2=0x7fdb980796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 -- 172.21.15.191:0/2069406615 shutdown_connections 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 -- 172.21.15.191:0/2069406615 wait complete. 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 Processor -- start 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.321+0000 7fdba01a4700 1 -- start start 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.322+0000 7fdba01a4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb9819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.322+0000 7fdba01a4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9810d310 con 0x7fdb98108e40 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.322+0000 7fdb9df40700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb9819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.322+0000 7fdb9df40700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb9819e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.191:47842/0 (socket says 172.21.15.191:47842) 2024-04-28T12:42:14.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.322+0000 7fdb9df40700 1 -- 172.21.15.191:0/1345715916 learned_addr learned my addr 172.21.15.191:0/1345715916 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:42:14.320 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.322+0000 7fdb9df40700 1 -- 172.21.15.191:0/1345715916 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9819eb20 con 0x7fdb98108e40 2024-04-28T12:42:14.320 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.322+0000 7fdb9df40700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb9819e5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb880056d0 tx=0x7fdb880057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:14.320 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.323+0000 7fdb8effd700 1 -- 172.21.15.191:0/1345715916 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb88026070 con 0x7fdb98108e40 2024-04-28T12:42:14.320 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.323+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9819ee10 con 0x7fdb98108e40 2024-04-28T12:42:14.320 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.323+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb98198880 con 0x7fdb98108e40 2024-04-28T12:42:14.320 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.323+0000 7fdb8effd700 1 -- 172.21.15.191:0/1345715916 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb8800b6f0 con 0x7fdb98108e40 2024-04-28T12:42:14.320 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.323+0000 7fdb8effd700 1 -- 172.21.15.191:0/1345715916 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb88030060 con 0x7fdb98108e40 2024-04-28T12:42:14.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.324+0000 7fdb8effd700 1 -- 172.21.15.191:0/1345715916 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fdb88029070 con 0x7fdb98108e40 2024-04-28T12:42:14.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.324+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7c005330 con 0x7fdb98108e40 2024-04-28T12:42:14.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.324+0000 7fdb8effd700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdb84039630 0x7fdb8403baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:14.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.324+0000 7fdb8effd700 1 -- 172.21.15.191:0/1345715916 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdb8801e070 con 0x7fdb98108e40 2024-04-28T12:42:14.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.324+0000 7fdb9d73f700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdb84039630 0x7fdb8403baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:14.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.325+0000 7fdb9d73f700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdb84039630 0x7fdb8403baf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdb94006fd0 tx=0x7fdb94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:14.325 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.329+0000 7fdb8effd700 1 -- 172.21.15.191:0/1345715916 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdb88013070 con 0x7fdb98108e40 2024-04-28T12:42:14.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:13 smithi053 ceph-mon[29182]: Updating smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:14.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:13 smithi053 ceph-mon[29182]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:14.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:13 smithi053 ceph-mon[29182]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:14.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:13 smithi053 ceph-mon[29182]: Updating smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:42:14.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:14.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:14.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:14.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.539+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi077:172.21.15.77=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}) -- 0x7fdb7c000c80 con 0x7fdb84039630 2024-04-28T12:42:14.541 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.546+0000 7fdb8effd700 1 -- 172.21.15.191:0/1345715916 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdb7c000c80 con 0x7fdb84039630 2024-04-28T12:42:14.541 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdb84039630 msgr2=0x7fdb8403baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdb84039630 0x7fdb8403baf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdb94006fd0 tx=0x7fdb94006e40 comp rx=0 tx=0).stop 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 msgr2=0x7fdb9819e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb9819e5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb880056d0 tx=0x7fdb880057b0 comp rx=0 tx=0).stop 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 shutdown_connections 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdb84039630 0x7fdb8403baf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 --2- 172.21.15.191:0/1345715916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb98108e40 0x7fdb9819e5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 >> 172.21.15.191:0/1345715916 conn(0x7fdb980792b0 msgr2=0x7fdb980feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 shutdown_connections 2024-04-28T12:42:14.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:14.548+0000 7fdba01a4700 1 -- 172.21.15.191:0/1345715916 wait complete. 2024-04-28T12:42:15.168 DEBUG:teuthology.orchestra.run.smithi077:mon.b> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mon.b.service 2024-04-28T12:42:15.171 DEBUG:teuthology.orchestra.run.smithi191:mon.c> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mon.c.service 2024-04-28T12:42:15.173 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T12:42:15.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T12:42:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi077:172.21.15.77=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi077:172.21.15.77=b;smithi191:172.21.15.191=c;count:3 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:15 smithi053 ceph-mon[29182]: Deploying daemon mon.c on smithi191 2024-04-28T12:42:16.356 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:17.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:17 smithi053 ceph-mon[29182]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.824+0000 7f42a2096700 1 -- 172.21.15.191:0/2468913257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 msgr2=0x7f429c1043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.824+0000 7f42a2096700 1 --2- 172.21.15.191:0/2468913257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c1043d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f428c009910 tx=0x7f428c009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.825+0000 7f42a2096700 1 -- 172.21.15.191:0/2468913257 shutdown_connections 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.825+0000 7f42a2096700 1 --2- 172.21.15.191:0/2468913257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c1043d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.825+0000 7f42a2096700 1 -- 172.21.15.191:0/2468913257 >> 172.21.15.191:0/2468913257 conn(0x7f429c0fd840 msgr2=0x7f429c0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.825+0000 7f42a2096700 1 -- 172.21.15.191:0/2468913257 shutdown_connections 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.825+0000 7f42a2096700 1 -- 172.21.15.191:0/2468913257 wait complete. 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.825+0000 7f42a2096700 1 Processor -- start 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.825+0000 7f42a2096700 1 -- start start 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.826+0000 7f42a2096700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:18.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.826+0000 7f42a2096700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f429c10d390 con 0x7f429c103ff0 2024-04-28T12:42:18.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.826+0000 7f429b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:18.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.826+0000 7f429b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.191:34190/0 (socket says 172.21.15.191:34190) 2024-04-28T12:42:18.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.826+0000 7f429b7fe700 1 -- 172.21.15.191:0/1632641439 learned_addr learned my addr 172.21.15.191:0/1632641439 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:42:18.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.826+0000 7f429b7fe700 1 -- 172.21.15.191:0/1632641439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c19ebd0 con 0x7f429c103ff0 2024-04-28T12:42:18.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.826+0000 7f429b7fe700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c19e690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f428c0056d0 tx=0x7f428c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:18.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.827+0000 7f4298ff9700 1 -- 172.21.15.191:0/1632641439 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c026070 con 0x7f429c103ff0 2024-04-28T12:42:18.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.827+0000 7f4298ff9700 1 -- 172.21.15.191:0/1632641439 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f428c004370 con 0x7f429c103ff0 2024-04-28T12:42:18.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.827+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429c19ee60 con 0x7f429c103ff0 2024-04-28T12:42:18.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.827+0000 7f4298ff9700 1 -- 172.21.15.191:0/1632641439 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c0039b0 con 0x7f429c103ff0 2024-04-28T12:42:18.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.827+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f429c198760 con 0x7f429c103ff0 2024-04-28T12:42:18.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.828+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f429c06bbb0 con 0x7f429c103ff0 2024-04-28T12:42:18.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.828+0000 7f4298ff9700 1 -- 172.21.15.191:0/1632641439 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f428c029070 con 0x7f429c103ff0 2024-04-28T12:42:18.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.828+0000 7f4298ff9700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4284039650 0x7f428403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:18.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.828+0000 7f4298ff9700 1 -- 172.21.15.191:0/1632641439 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f428c01e070 con 0x7f429c103ff0 2024-04-28T12:42:18.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.828+0000 7f429affd700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4284039650 0x7f428403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:18.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.829+0000 7f429affd700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4284039650 0x7f428403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4290006fd0 tx=0x7f4290006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:18.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:18.833+0000 7f4298ff9700 1 -- 172.21.15.191:0/1632641439 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f428c013070 con 0x7f429c103ff0 2024-04-28T12:42:19.133 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:19 smithi191 systemd[1]: Starting Ceph mon.c for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:42:19.134 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.136+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f429c064a80 con 0x7f429c103ff0 2024-04-28T12:42:19.134 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.136+0000 7f4298ff9700 1 -- 172.21.15.191:0/1632641439 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f428c030020 con 0x7f429c103ff0 2024-04-28T12:42:19.134 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:42:19.134 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"10525726-055c-11ef-bc93-c7b262605968","modified":"2024-04-28T12:40:35.008027Z","created":"2024-04-28T12:40:35.008027Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4284039650 msgr2=0x7f428403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4284039650 0x7f428403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4290006fd0 tx=0x7f4290006e40 comp rx=0 tx=0).stop 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 msgr2=0x7f429c19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c19e690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f428c0056d0 tx=0x7f428c0057b0 comp rx=0 tx=0).stop 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 shutdown_connections 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4284039650 0x7f428403bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 --2- 172.21.15.191:0/1632641439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f429c103ff0 0x7f429c19e690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 >> 172.21.15.191:0/1632641439 conn(0x7f429c0fd840 msgr2=0x7f429c106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 shutdown_connections 2024-04-28T12:42:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:19.139+0000 7f42a2096700 1 -- 172.21.15.191:0/1632641439 wait complete. 2024-04-28T12:42:19.137 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-04-28T12:42:19.716 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:19 smithi191 podman[32994]: 2024-04-28 12:42:19.361327363 +0000 UTC m=+0.019056721 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:42:19.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:19 smithi053 ceph-mon[29182]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:19.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:19 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/1632641439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:42:19.985 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:19 smithi191 podman[32994]: 2024-04-28 12:42:19.879214663 +0000 UTC m=+0.536943995 container create 1490b39f630f1ef1ac41b364001085c2d6092b6ab2b13ddc3f6524bfd448b120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-28T12:42:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 podman[32994]: 2024-04-28 12:42:20.098646081 +0000 UTC m=+0.756375422 container init 1490b39f630f1ef1ac41b364001085c2d6092b6ab2b13ddc3f6524bfd448b120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-28T12:42:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 podman[32994]: 2024-04-28 12:42:20.101520965 +0000 UTC m=+0.759250307 container start 1490b39f630f1ef1ac41b364001085c2d6092b6ab2b13ddc3f6524bfd448b120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-28T12:42:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: set uid:gid to 167:167 (ceph:ceph) 2024-04-28T12:42:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-28T12:42:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pidfile_write: ignore empty --pid-file 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: load: jerasure load: lrc 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: RocksDB version: 7.9.2 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Git sha 0 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: DB SUMMARY 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: DB Session ID: 0285Y89UIRHVHYPOANZD 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: CURRENT file: CURRENT 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: IDENTITY file: IDENTITY 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.error_if_exists: 0 2024-04-28T12:42:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.create_if_missing: 0 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.paranoid_checks: 1 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.env: 0x55eccb6b6cc0 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.fs: PosixFileSystem 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.info_log: 0x55eccc6f5d80 2024-04-28T12:42:20.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_file_opening_threads: 16 2024-04-28T12:42:20.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.statistics: (nil) 2024-04-28T12:42:20.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.use_fsync: 0 2024-04-28T12:42:20.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_log_file_size: 0 2024-04-28T12:42:20.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-28T12:42:20.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-28T12:42:20.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.keep_log_file_num: 1000 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.recycle_log_file_num: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.allow_fallocate: 1 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.allow_mmap_reads: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.allow_mmap_writes: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.use_direct_reads: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.create_missing_column_families: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.db_log_dir: 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.wal_dir: 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-28T12:42:20.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.advise_random_on_open: 1 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.db_write_buffer_size: 0 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.write_buffer_manager: 0x55eccc6c70e0 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.rate_limiter: (nil) 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.wal_recovery_mode: 2 2024-04-28T12:42:20.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.enable_thread_tracking: 0 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.enable_pipelined_write: 0 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.unordered_write: 0 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.row_cache: None 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.wal_filter: None 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.allow_ingest_behind: 0 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.two_write_queues: 0 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.manual_wal_flush: 0 2024-04-28T12:42:20.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.wal_compression: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.atomic_flush: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.log_readahead_size: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.best_efforts_recovery: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.allow_data_in_errors: 0 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.db_host_id: __hostname__ 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_background_jobs: 2 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_background_compactions: -1 2024-04-28T12:42:20.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_subcompactions: 1 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_total_wal_size: 0 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_open_files: -1 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bytes_per_sync: 0 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_readahead_size: 0 2024-04-28T12:42:20.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_background_flushes: -1 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Compression algorithms supported: 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kZSTD supported: 0 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kXpressCompression supported: 0 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kLZ4HCCompression supported: 1 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kZlibCompression supported: 1 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kSnappyCompression supported: 1 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kLZ4Compression supported: 1 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: kBZip2Compression supported: 0 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-28T12:42:20.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.merge_operator: 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_filter: None 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_filter_factory: None 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.sst_partitioner_factory: None 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eccc6f5f40) 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks: 1 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_top_level_index_and_filter: 1 2024-04-28T12:42:20.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_type: 0 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_index_type: 0 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_shortening: 1 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: checksum: 4 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: no_block_cache: 0 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache: 0x55eccc6f91f0 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_name: BinnedLRUCache 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_options: 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: capacity : 536870912 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_shard_bits : 4 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: strict_capacity_limit : 0 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: high_pri_pool_ratio: 0.000 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_compressed: (nil) 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: persistent_cache: (nil) 2024-04-28T12:42:20.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size: 4096 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size_deviation: 10 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_restart_interval: 16 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_block_restart_interval: 1 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: metadata_block_size: 4096 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: partition_filters: 0 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: use_delta_encoding: 1 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: filter_policy: bloomfilter 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: whole_key_filtering: 1 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: verify_compression: 0 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: read_amp_bytes_per_bit: 0 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: format_version: 5 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_index_compression: 1 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_align: 0 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: max_auto_readahead_size: 262144 2024-04-28T12:42:20.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: prepopulate_block_cache: 0 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout: initial_auto_readahead_size: 8192 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_file_reads_for_auto_readahead: 2 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.write_buffer_size: 33554432 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_write_buffer_number: 2 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression: NoCompression 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression: Disabled 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.prefix_extractor: nullptr 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.num_levels: 7 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-28T12:42:20.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.level: 32767 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.strategy: 0 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.enabled: false 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-28T12:42:20.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.target_file_size_base: 67108864 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-28T12:42:20.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.arena_block_size: 1048576 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.disable_auto_compactions: 0 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-28T12:42:20.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.inplace_update_support: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.bloom_locality: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.max_successive_merges: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.paranoid_file_checks: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.force_consistency_checks: 1 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.report_bg_io_stats: 0 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.ttl: 2592000 2024-04-28T12:42:20.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-28T12:42:20.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-28T12:42:20.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.enable_blob_files: false 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.min_blob_size: 0 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.blob_file_size: 268435456 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.blob_file_starting_level: 0 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7cd9df3a-c1c0-463a-8050-389a7c423678 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308140142816, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-28T12:42:20.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308140143218, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714308140, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7cd9df3a-c1c0-463a-8050-389a7c423678", "db_session_id": "0285Y89UIRHVHYPOANZD", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308140143297, "job": 1, "event": "recovery_finished"} 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eccc816000 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: DB pointer 0x55eccc802000 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** DB Stats ** 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:42:20.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-28T12:42:20.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache BinnedLRUCache@0x55eccc6f91f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-04-28T12:42:20.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: using public_addr v2:172.21.15.191:0/0 -> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: starting mon.c rank -1 at public addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] at bind addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(???) e0 preinit fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 bash[32994]: 1490b39f630f1ef1ac41b364001085c2d6092b6ab2b13ddc3f6524bfd448b120 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 systemd[1]: Started Ceph mon.c for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).mds e1 new map 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: e1 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: legacy client fscid: -1 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: No filesystems configured 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-28T12:42:20.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Added host smithi077 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Deploying cephadm binary to smithi191 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:20.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Added host smithi191 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/706258250' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-28T12:42:20.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/706258250' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Updating smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi077:172.21.15.77=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi077:172.21.15.77=b;smithi191:172.21.15.191=c;count:3 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: Deploying daemon mon.c on smithi191 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/1632641439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:42:20.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:20 smithi191 ceph-mon[33016]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-28T12:42:20.743 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T12:42:20.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T12:42:20.975 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:42:25.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.422+0000 7fc278853700 1 -- 172.21.15.191:0/1535472872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25c0035c0 msgr2=0x7fc25c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.422+0000 7fc278853700 1 --2- 172.21.15.191:0/1535472872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25c0035c0 0x7fc25c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc27406ae20 tx=0x7fc26400feb0 comp rx=0 tx=0).stop 2024-04-28T12:42:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.422+0000 7fc278853700 1 -- 172.21.15.191:0/1535472872 shutdown_connections 2024-04-28T12:42:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.423+0000 7fc278853700 1 --2- 172.21.15.191:0/1535472872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25c0035c0 0x7fc25c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.423+0000 7fc278853700 1 -- 172.21.15.191:0/1535472872 >> 172.21.15.191:0/1535472872 conn(0x7fc2740fd900 msgr2=0x7fc2740ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.423+0000 7fc278853700 1 -- 172.21.15.191:0/1535472872 shutdown_connections 2024-04-28T12:42:25.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.423+0000 7fc278853700 1 -- 172.21.15.191:0/1535472872 wait complete. 2024-04-28T12:42:25.421 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.424+0000 7fc278853700 1 Processor -- start 2024-04-28T12:42:25.421 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.424+0000 7fc278853700 1 -- start start 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.424+0000 7fc278853700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 0x7fc27419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.424+0000 7fc278853700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc27419ec40 0x7fc274198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.424+0000 7fc278853700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc27410ae50 con 0x7fc27419ec40 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.424+0000 7fc278853700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc27410acd0 con 0x7fc25c0035c0 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc27259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 0x7fc27419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc27259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 0x7fc27419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:54586/0 (socket says 172.21.15.191:54586) 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc27259c700 1 -- 172.21.15.191:0/1268773083 learned_addr learned my addr 172.21.15.191:0/1268773083 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc27259c700 1 -- 172.21.15.191:0/1268773083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 msgr2=0x7fc27419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-28T12:42:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc27259c700 1 -- 172.21.15.191:0/1268773083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 msgr2=0x7fc27419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-28T12:42:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc27259c700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 0x7fc27419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-28T12:42:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc27259c700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 0x7fc27419e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:42:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc271d9b700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc27419ec40 0x7fc274198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc271d9b700 1 -- 172.21.15.191:0/1268773083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 msgr2=0x7fc27419e700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:42:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc271d9b700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 0x7fc27419e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc271d9b700 1 -- 172.21.15.191:0/1268773083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27419f280 con 0x7fc27419ec40 2024-04-28T12:42:25.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.425+0000 7fc271d9b700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc27419ec40 0x7fc274198910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc26800b4d0 tx=0x7fc26800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.426+0000 7fc2637fe700 1 -- 172.21.15.191:0/1268773083 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc268011630 con 0x7fc27419ec40 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.426+0000 7fc2637fe700 1 -- 172.21.15.191:0/1268773083 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc268011c80 con 0x7fc27419ec40 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.426+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc274198e50 con 0x7fc27419ec40 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.426+0000 7fc2637fe700 1 -- 172.21.15.191:0/1268773083 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc268010da0 con 0x7fc27419ec40 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.426+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2741993e0 con 0x7fc27419ec40 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.427+0000 7fc2637fe700 1 -- 172.21.15.191:0/1268773083 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc268018020 con 0x7fc27419ec40 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.427+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc27406b620 con 0x7fc27419ec40 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.427+0000 7fc2637fe700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc2540393f0 0x7fc25403b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:25.424 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.427+0000 7fc2637fe700 1 -- 172.21.15.191:0/1268773083 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc26804c300 con 0x7fc27419ec40 2024-04-28T12:42:25.425 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.428+0000 7fc27259c700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc2540393f0 0x7fc25403b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:25.425 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.428+0000 7fc27259c700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc2540393f0 0x7fc25403b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc26400fa60 tx=0x7fc264005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:25.430 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.433+0000 7fc2637fe700 1 -- 172.21.15.191:0/1268773083 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc26801f020 con 0x7fc27419ec40 2024-04-28T12:42:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:42:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: mon.a calling monitor election 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: mon.c calling monitor election 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: monmap epoch 2 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: last_changed 2024-04-28T12:42:20.167136+0000 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: min_mon_release 19 (squid) 2024-04-28T12:42:25.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: election_strategy: 1 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: fsmap 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: mgrmap e13: x(active, since 50s) 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: overall HEALTH_OK 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:25.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:42:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: mon.a calling monitor election 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: mon.c calling monitor election 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: monmap epoch 2 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: last_changed 2024-04-28T12:42:20.167136+0000 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: min_mon_release 19 (squid) 2024-04-28T12:42:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: election_strategy: 1 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: fsmap 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: mgrmap e13: x(active, since 50s) 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: overall HEALTH_OK 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:25.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:25.719 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.723+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc274199df0 con 0x7fc27419ec40 2024-04-28T12:42:25.720 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.723+0000 7fc2637fe700 1 -- 172.21.15.191:0/1268773083 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fc268019d70 con 0x7fc27419ec40 2024-04-28T12:42:25.720 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:42:25.720 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"10525726-055c-11ef-bc93-c7b262605968","modified":"2024-04-28T12:42:20.167136Z","created":"2024-04-28T12:40:35.008027Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-28T12:42:25.722 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc2540393f0 msgr2=0x7fc25403b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:25.722 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc2540393f0 0x7fc25403b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc26400fa60 tx=0x7fc264005c20 comp rx=0 tx=0).stop 2024-04-28T12:42:25.722 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc27419ec40 msgr2=0x7fc274198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:25.722 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc27419ec40 0x7fc274198910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc26800b4d0 tx=0x7fc26800b890 comp rx=0 tx=0).stop 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 shutdown_connections 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc2540393f0 0x7fc25403b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc25c0035c0 0x7fc27419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 --2- 172.21.15.191:0/1268773083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc27419ec40 0x7fc274198910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.725+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 >> 172.21.15.191:0/1268773083 conn(0x7fc2740fd900 msgr2=0x7fc274104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.726+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 shutdown_connections 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:25.726+0000 7fc278853700 1 -- 172.21.15.191:0/1268773083 wait complete. 2024-04-28T12:42:25.723 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-04-28T12:42:26.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:26 smithi191 ceph-mon[33016]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:26.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:26 smithi191 ceph-mon[33016]: Deploying daemon mon.b on smithi077 2024-04-28T12:42:26.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:26 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/1268773083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:42:26.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:26 smithi053 ceph-mon[29182]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:26 smithi053 ceph-mon[29182]: Deploying daemon mon.b on smithi077 2024-04-28T12:42:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:26 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/1268773083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:42:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:27.349 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T12:42:27.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T12:42:27.583 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:42:28.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:28 smithi191 ceph-mon[33016]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:28 smithi053 ceph-mon[29182]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.034+0000 7f0deff6a700 1 -- 172.21.15.191:0/478809382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de8103190 msgr2=0x7f0de8103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.034+0000 7f0deff6a700 1 --2- 172.21.15.191:0/478809382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de8103190 0x7f0de8103570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0de4009910 tx=0x7f0de4009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.034+0000 7f0deff6a700 1 -- 172.21.15.191:0/478809382 shutdown_connections 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.034+0000 7f0deff6a700 1 --2- 172.21.15.191:0/478809382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de8103190 0x7f0de8103570 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.034+0000 7f0deff6a700 1 -- 172.21.15.191:0/478809382 >> 172.21.15.191:0/478809382 conn(0x7f0de80fea20 msgr2=0x7f0de8100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.034+0000 7f0deff6a700 1 -- 172.21.15.191:0/478809382 shutdown_connections 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.034+0000 7f0deff6a700 1 -- 172.21.15.191:0/478809382 wait complete. 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0deff6a700 1 Processor -- start 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0deff6a700 1 -- start start 2024-04-28T12:42:29.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0deff6a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 0x7f0de819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0deff6a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de819ecb0 0x7f0de8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0deff6a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0de810d410 con 0x7f0de819ecb0 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0deff6a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0de810d290 con 0x7f0de8103190 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0ded505700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de819ecb0 0x7f0de8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0ded505700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de819ecb0 0x7f0de8198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.191:57796/0 (socket says 172.21.15.191:57796) 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0ded505700 1 -- 172.21.15.191:0/1814417749 learned_addr learned my addr 172.21.15.191:0/1814417749 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0dedd06700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 0x7f0de819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0dedd06700 1 -- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 msgr2=0x7f0de819e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0ded505700 1 -- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 msgr2=0x7f0de819e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0dedd06700 1 -- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 msgr2=0x7f0de819e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0dedd06700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 0x7f0de819e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-28T12:42:29.034 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0dedd06700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 0x7f0de819e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0ded505700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 0x7f0de819e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.035+0000 7f0ded505700 1 -- 172.21.15.191:0/1814417749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de819f2f0 con 0x7f0de819ecb0 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.036+0000 7f0ded505700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de819ecb0 0x7f0de8198930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800b700 tx=0x7f0dd800bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.036+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0dd8011630 con 0x7f0de819ecb0 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.036+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dd8011c80 con 0x7f0de819ecb0 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.036+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de8198e70 con 0x7f0de819ecb0 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.036+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0dd8010da0 con 0x7f0de819ecb0 2024-04-28T12:42:29.035 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.036+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de8199400 con 0x7f0de819ecb0 2024-04-28T12:42:29.036 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.037+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0dd8018020 con 0x7f0de819ecb0 2024-04-28T12:42:29.036 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.037+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de8104670 con 0x7f0de819ecb0 2024-04-28T12:42:29.036 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.037+0000 7f0ddeffd700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0dd40393f0 0x7f0dd403b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:29.036 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.037+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0dd804c3d0 con 0x7f0de819ecb0 2024-04-28T12:42:29.036 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.037+0000 7f0dedd06700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0dd40393f0 0x7f0dd403b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:29.036 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.038+0000 7f0dedd06700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0dd40393f0 0x7f0dd403b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0de400b5c0 tx=0x7f0de401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:29.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.043+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0dd801f020 con 0x7f0de819ecb0 2024-04-28T12:42:29.152 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.156+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80177d0 con 0x7f0de819ecb0 2024-04-28T12:42:29.172 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:29 smithi077 ceph-mon[32553]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-28T12:42:29.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:29.325+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0de8199f00 con 0x7f0de819ecb0 2024-04-28T12:42:29.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:29 smithi077 bash[32534]: 49b5bb26e08958be2dd70b2aaff7f092926282e6c5b1c08ec3a2fa18b6450981 2024-04-28T12:42:29.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:29 smithi077 systemd[1]: Started Ceph mon.b for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:42:34.161 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.164+0000 7f0ddeffd700 1 -- 172.21.15.191:0/1814417749 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f0dd803e510 con 0x7f0de819ecb0 2024-04-28T12:42:34.162 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-28T12:42:34.162 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"10525726-055c-11ef-bc93-c7b262605968","modified":"2024-04-28T12:42:29.156518Z","created":"2024-04-28T12:40:35.008027Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-28T12:42:34.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.167+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0dd40393f0 msgr2=0x7f0dd403b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:34.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.167+0000 7f0deff6a700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0dd40393f0 0x7f0dd403b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0de400b5c0 tx=0x7f0de401a040 comp rx=0 tx=0).stop 2024-04-28T12:42:34.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.167+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de819ecb0 msgr2=0x7f0de8198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:34.164 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.167+0000 7f0deff6a700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de819ecb0 0x7f0de8198930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800b700 tx=0x7f0dd800bac0 comp rx=0 tx=0).stop 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.168+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 shutdown_connections 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.168+0000 7f0deff6a700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0dd40393f0 0x7f0dd403b8b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.168+0000 7f0deff6a700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de8103190 0x7f0de819e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.168+0000 7f0deff6a700 1 --2- 172.21.15.191:0/1814417749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0de819ecb0 0x7f0de8198930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.168+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 >> 172.21.15.191:0/1814417749 conn(0x7f0de80fea20 msgr2=0x7f0de8104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.168+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 shutdown_connections 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:34.168+0000 7f0deff6a700 1 -- 172.21.15.191:0/1814417749 wait complete. 2024-04-28T12:42:34.165 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-04-28T12:42:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: mon.c calling monitor election 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: mon.a calling monitor election 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/1814417749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: monmap epoch 3 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: last_changed 2024-04-28T12:42:29.156518+0000 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: min_mon_release 19 (squid) 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: election_strategy: 1 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: fsmap 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: mgrmap e13: x(active, since 59s) 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: overall HEALTH_OK 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:34.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: mon.c calling monitor election 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: mon.a calling monitor election 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/1814417749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: monmap epoch 3 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: last_changed 2024-04-28T12:42:29.156518+0000 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: min_mon_release 19 (squid) 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: election_strategy: 1 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: fsmap 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: mgrmap e13: x(active, since 59s) 2024-04-28T12:42:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: overall HEALTH_OK 2024-04-28T12:42:34.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:34.883 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-28T12:42:34.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-28T12:42:35.140 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:42:35.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:35.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:35.542 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:35.542 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:35.542 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:36.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:42:36.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.c calling monitor election 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.a calling monitor election 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/1814417749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:42:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: monmap epoch 3 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: last_changed 2024-04-28T12:42:29.156518+0000 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: min_mon_release 19 (squid) 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: election_strategy: 1 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: fsmap 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mgrmap e13: x(active, since 59s) 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T12:42:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:36.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.761+0000 7f35ea226700 1 -- 172.21.15.53:0/1975329993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e4108ec0 msgr2=0x7f35e41092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.761+0000 7f35ea226700 1 --2- 172.21.15.53:0/1975329993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e4108ec0 0x7f35e41092a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009910 tx=0x7f35d4009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.761+0000 7f35ea226700 1 -- 172.21.15.53:0/1975329993 shutdown_connections 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.761+0000 7f35ea226700 1 --2- 172.21.15.53:0/1975329993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e4108ec0 0x7f35e41092a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.761+0000 7f35ea226700 1 -- 172.21.15.53:0/1975329993 >> 172.21.15.53:0/1975329993 conn(0x7f35e4077880 msgr2=0x7f35e4077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.761+0000 7f35ea226700 1 -- 172.21.15.53:0/1975329993 shutdown_connections 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.761+0000 7f35ea226700 1 -- 172.21.15.53:0/1975329993 wait complete. 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 Processor -- start 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 -- start start 2024-04-28T12:42:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35e4108ec0 0x7f35e41a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e41a0e40 0x7f35e419aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e419b000 0x7f35e419b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410d300 con 0x7f35e419b000 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410d180 con 0x7f35e41a0e40 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.762+0000 7f35ea226700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410d480 con 0x7f35e4108ec0 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e419b000 0x7f35e419b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e419b000 0x7f35e419b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52198/0 (socket says 172.21.15.53:52198) 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 -- 172.21.15.53:0/3249229678 learned_addr learned my addr 172.21.15.53:0/3249229678 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e37fe700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35e4108ec0 0x7f35e41a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e2ffd700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e41a0e40 0x7f35e419aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 -- 172.21.15.53:0/3249229678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35e4108ec0 msgr2=0x7f35e41a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35e4108ec0 0x7f35e41a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 -- 172.21.15.53:0/3249229678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e41a0e40 msgr2=0x7f35e419aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e41a0e40 0x7f35e419aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 -- 172.21.15.53:0/3249229678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e41a77d0 con 0x7f35e419b000 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e3fff700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e419b000 0x7f35e419b4b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00ba50 tx=0x7f35dc00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e0ff9700 1 -- 172.21.15.53:0/3249229678 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc019070 con 0x7f35e419b000 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e0ff9700 1 -- 172.21.15.53:0/3249229678 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35dc00cde0 con 0x7f35e419b000 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e41a7a00 con 0x7f35e419b000 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35e0ff9700 1 -- 172.21.15.53:0/3249229678 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc007350 con 0x7f35e419b000 2024-04-28T12:42:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.763+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e41a7f90 con 0x7f35e419b000 2024-04-28T12:42:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.764+0000 7f35e0ff9700 1 -- 172.21.15.53:0/3249229678 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f35dc01c070 con 0x7f35e419b000 2024-04-28T12:42:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.764+0000 7f35e0ff9700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f35cc039ae0 0x7f35cc03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.764+0000 7f35e0ff9700 1 -- 172.21.15.53:0/3249229678 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f35dc04cdc0 con 0x7f35e419b000 2024-04-28T12:42:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.765+0000 7f35e37fe700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f35cc039ae0 0x7f35cc03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.765+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e4064a80 con 0x7f35e419b000 2024-04-28T12:42:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.765+0000 7f35e37fe700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f35cc039ae0 0x7f35cc03bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35d4006010 tx=0x7f35d401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:36.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.770+0000 7f35e0ff9700 1 -- 172.21.15.53:0/3249229678 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f35dc00e0a0 con 0x7f35e419b000 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: mon.b calling monitor election 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: Updating smithi053:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: Updating smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: Updating smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: mon.b calling monitor election 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: mon.c calling monitor election 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: mon.a calling monitor election 2024-04-28T12:42:36.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: monmap epoch 3 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: last_changed 2024-04-28T12:42:29.156518+0000 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: min_mon_release 19 (squid) 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: election_strategy: 1 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: fsmap 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: mgrmap e13: x(active, since 61s) 2024-04-28T12:42:36.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: overall HEALTH_OK 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:36.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.b calling monitor election 2024-04-28T12:42:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-28T12:42:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-28T12:42:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-28T12:42:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: Updating smithi053:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: Updating smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: Updating smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.b calling monitor election 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.c calling monitor election 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.a calling monitor election 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: monmap epoch 3 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: last_changed 2024-04-28T12:42:29.156518+0000 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: min_mon_release 19 (squid) 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: election_strategy: 1 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: fsmap 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:36.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: mgrmap e13: x(active, since 61s) 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:36.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:36.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:36.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.991+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f35e402e490 con 0x7f35e419b000 2024-04-28T12:42:36.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.991+0000 7f35e0ff9700 1 -- 172.21.15.53:0/3249229678 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f35dc01c520 con 0x7f35e419b000 2024-04-28T12:42:36.987 INFO:teuthology.orchestra.run.smithi053.stdout:# minimal ceph.conf for 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:36.987 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-04-28T12:42:36.987 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:36.987 INFO:teuthology.orchestra.run.smithi053.stdout: mon_host = [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-04-28T12:42:36.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f35cc039ae0 msgr2=0x7f35cc03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:36.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f35cc039ae0 0x7f35cc03bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35d4006010 tx=0x7f35d401a040 comp rx=0 tx=0).stop 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e419b000 msgr2=0x7f35e419b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e419b000 0x7f35e419b4b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00ba50 tx=0x7f35dc00bd60 comp rx=0 tx=0).stop 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 shutdown_connections 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f35cc039ae0 0x7f35cc03bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e419b000 0x7f35e419b4b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35e4108ec0 0x7f35e41a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 --2- 172.21.15.53:0/3249229678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e41a0e40 0x7f35e419aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 >> 172.21.15.53:0/3249229678 conn(0x7f35e4077880 msgr2=0x7f35e40ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 shutdown_connections 2024-04-28T12:42:36.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:36.993+0000 7f35ea226700 1 -- 172.21.15.53:0/3249229678 wait complete. 2024-04-28T12:42:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: mon.b calling monitor election 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: Updating smithi053:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: Updating smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: Updating smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: mon.b calling monitor election 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: mon.c calling monitor election 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: mon.a calling monitor election 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: monmap epoch 3 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: last_changed 2024-04-28T12:42:29.156518+0000 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: created 2024-04-28T12:40:35.008027+0000 2024-04-28T12:42:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: min_mon_release 19 (squid) 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: election_strategy: 1 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: 2: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: fsmap 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: mgrmap e13: x(active, since 61s) 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: overall HEALTH_OK 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:37.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:37.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:37.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:37 smithi053 ceph-mon[29182]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:37 smithi053 ceph-mon[29182]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T12:42:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:37 smithi053 ceph-mon[29182]: Reconfiguring daemon mon.a on smithi053 2024-04-28T12:42:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:37 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3249229678' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:37.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:37 smithi077 ceph-mon[32553]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:37 smithi077 ceph-mon[32553]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T12:42:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:37 smithi077 ceph-mon[32553]: Reconfiguring daemon mon.a on smithi053 2024-04-28T12:42:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:37 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3249229678' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:37.959 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-28T12:42:37.960 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:42:37.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T12:42:37.990 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:42:37.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:37 smithi191 ceph-mon[33016]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:37 smithi191 ceph-mon[33016]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T12:42:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:37 smithi191 ceph-mon[33016]: Reconfiguring daemon mon.a on smithi053 2024-04-28T12:42:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:37 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3249229678' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:42:38.057 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:42:38.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T12:42:38.088 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:42:38.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:38.155 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:42:38.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T12:42:38.186 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:42:38.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:42:38.254 INFO:tasks.cephadm:Adding mgr.x on smithi053 2024-04-28T12:42:38.254 INFO:tasks.cephadm:Adding mgr.y on smithi077 2024-04-28T12:42:38.254 INFO:tasks.cephadm:Adding mgr.z on smithi191 2024-04-28T12:42:38.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi053=x;smithi077=y;smithi191=z' 2024-04-28T12:42:38.531 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 -- 172.21.15.191:0/3985700178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364102fb0 msgr2=0x7fb3641033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3985700178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364102fb0 0x7fb3641033d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb358009910 tx=0x7fb358009c20 comp rx=0 tx=0).stop 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 -- 172.21.15.191:0/3985700178 shutdown_connections 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3985700178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364102fb0 0x7fb3641033d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 -- 172.21.15.191:0/3985700178 >> 172.21.15.191:0/3985700178 conn(0x7fb3640fe550 msgr2=0x7fb364100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 -- 172.21.15.191:0/3985700178 shutdown_connections 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 -- 172.21.15.191:0/3985700178 wait complete. 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 Processor -- start 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.004+0000 7fb36aa37700 1 -- start start 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb36aa37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb364102fb0 0x7fb3640771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:40.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb36aa37700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364075810 0x7fb364075c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb36aa37700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3640761d0 0x7fb3641a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb36aa37700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb364108ef0 con 0x7fb364075810 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb36aa37700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb364108d70 con 0x7fb364102fb0 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb36aa37700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb364109070 con 0x7fb3640761d0 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3640761d0 0x7fb3641a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3640761d0 0x7fb3641a2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:37084/0 (socket says 172.21.15.191:37084) 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 -- 172.21.15.191:0/3418829999 learned_addr learned my addr 172.21.15.191:0/3418829999 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb3637fe700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364075810 0x7fb364075c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb363fff700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb364102fb0 0x7fb3640771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 -- 172.21.15.191:0/3418829999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb364102fb0 msgr2=0x7fb3640771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb364102fb0 0x7fb3640771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 -- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364075810 msgr2=0x7fb364075c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364075810 0x7fb364075c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:40.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 -- 172.21.15.191:0/3418829999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3641a2d20 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.005+0000 7fb368fd4700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3640761d0 0x7fb3641a2760 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00ba50 tx=0x7fb35c00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.006+0000 7fb3617fa700 1 -- 172.21.15.191:0/3418829999 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c019070 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.006+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3641a2fb0 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.006+0000 7fb3617fa700 1 -- 172.21.15.191:0/3418829999 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb35c014440 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.006+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3641a3540 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.006+0000 7fb3617fa700 1 -- 172.21.15.191:0/3418829999 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00ce60 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.007+0000 7fb3617fa700 1 -- 172.21.15.191:0/3418829999 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb35c01c070 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.007+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb350005360 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.007+0000 7fb3617fa700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb34c039ae0 0x7fb34c03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.007+0000 7fb3617fa700 1 -- 172.21.15.191:0/3418829999 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb35c04bd20 con 0x7fb3640761d0 2024-04-28T12:42:40.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.007+0000 7fb363fff700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb34c039ae0 0x7fb34c03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:40.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.008+0000 7fb363fff700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb34c039ae0 0x7fb34c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb3580050f0 tx=0x7fb35801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:40.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.012+0000 7fb3617fa700 1 -- 172.21.15.191:0/3418829999 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb35c012500 con 0x7fb3640761d0 2024-04-28T12:42:40.031 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:40.032 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:40.032 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:40.032 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: Reconfiguring mon.b (monmap changed)... 2024-04-28T12:42:40.032 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:40.032 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:40.032 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:40.032 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:39 smithi077 ceph-mon[32553]: Reconfiguring daemon mon.b on smithi077 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: Reconfiguring mon.b (monmap changed)... 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:39 smithi191 ceph-mon[33016]: Reconfiguring daemon mon.b on smithi077 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: Reconfiguring mon.b (monmap changed)... 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:39 smithi053 ceph-mon[29182]: Reconfiguring daemon mon.b on smithi077 2024-04-28T12:42:40.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.223+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi077=y;smithi191=z", "target": ["mon-mgr", ""]}) -- 0x7fb350000bf0 con 0x7fb34c039ae0 2024-04-28T12:42:40.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.230+0000 7fb3617fa700 1 -- 172.21.15.191:0/3418829999 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb350000bf0 con 0x7fb34c039ae0 2024-04-28T12:42:40.227 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-04-28T12:42:40.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb34c039ae0 msgr2=0x7fb34c03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:40.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb34c039ae0 0x7fb34c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb3580050f0 tx=0x7fb35801a040 comp rx=0 tx=0).stop 2024-04-28T12:42:40.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3640761d0 msgr2=0x7fb3641a2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:40.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3640761d0 0x7fb3641a2760 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00ba50 tx=0x7fb35c00bd60 comp rx=0 tx=0).stop 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 shutdown_connections 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb34c039ae0 0x7fb34c03bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3640761d0 0x7fb3641a2760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb364102fb0 0x7fb3640771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 --2- 172.21.15.191:0/3418829999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb364075810 0x7fb364075c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.232+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 >> 172.21.15.191:0/3418829999 conn(0x7fb3640fe550 msgr2=0x7fb3640ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.233+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 shutdown_connections 2024-04-28T12:42:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:42:40.233+0000 7fb36aa37700 1 -- 172.21.15.191:0/3418829999 wait complete. 2024-04-28T12:42:40.871 DEBUG:teuthology.orchestra.run.smithi077:mgr.y> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.y.service 2024-04-28T12:42:40.874 DEBUG:teuthology.orchestra.run.smithi191:mgr.z> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.z.service 2024-04-28T12:42:40.877 INFO:tasks.cephadm:Deploying OSDs... 2024-04-28T12:42:40.877 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:42:40.877 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T12:42:40.897 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T12:42:40.897 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-04-28T12:42:40.956 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T12:42:40.956 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:40.956 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-28T12:42:40.957 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:40.957 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:40.957 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-28 12:41:46.102012770 +0000 2024-04-28T12:42:40.957 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-28 12:34:40.270406628 +0000 2024-04-28T12:42:40.957 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-28 12:34:40.270406628 +0000 2024-04-28T12:42:40.957 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-28 12:34:40.270406628 +0000 2024-04-28T12:42:40.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T12:42:41.025 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-28T12:42:41.025 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-28T12:42:41.025 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000114448 s, 4.5 MB/s 2024-04-28T12:42:41.026 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T12:42:41.085 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-28 12:41:46.131012235 +0000 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-28 12:34:40.598400313 +0000 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-28 12:34:40.598400313 +0000 2024-04-28T12:42:41.143 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-28 12:34:40.598400313 +0000 2024-04-28T12:42:41.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T12:42:41.208 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-28T12:42:41.208 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-28T12:42:41.208 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000111622 s, 4.6 MB/s 2024-04-28T12:42:41.209 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T12:42:41.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi077=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:41.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: Saving service mgr spec with placement smithi053=x;smithi077=y;smithi191=z;count:3 2024-04-28T12:42:41.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:41.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:41.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:41.266 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-04-28T12:42:41.322 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T12:42:41.322 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:41.322 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-28T12:42:41.322 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:41.322 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:41.322 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-28 12:41:46.161011683 +0000 2024-04-28T12:42:41.322 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-28 12:34:40.937393786 +0000 2024-04-28T12:42:41.323 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-28 12:34:40.937393786 +0000 2024-04-28T12:42:41.323 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-28 12:34:40.937393786 +0000 2024-04-28T12:42:41.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T12:42:41.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi077=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:41.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: Saving service mgr spec with placement smithi053=x;smithi077=y;smithi191=z;count:3 2024-04-28T12:42:41.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:41.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:41.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-28T12:42:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-28T12:42:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.4878e-05 s, 5.4 MB/s 2024-04-28T12:42:41.389 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T12:42:41.447 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-28 12:41:46.198011001 +0000 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-28 12:34:41.275387278 +0000 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-28 12:34:41.275387278 +0000 2024-04-28T12:42:41.503 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-28 12:34:41.275387278 +0000 2024-04-28T12:42:41.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T12:42:41.568 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-28T12:42:41.568 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-28T12:42:41.568 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000112719 s, 4.5 MB/s 2024-04-28T12:42:41.569 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T12:42:41.626 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-28T12:42:41.627 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T12:42:41.645 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T12:42:41.645 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi077=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: Saving service mgr spec with placement smithi053=x;smithi077=y;smithi191=z;count:3 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:42:41.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:41.703 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-28 12:42:28.612006460 +0000 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-28 12:34:58.774323428 +0000 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-28 12:34:58.774323428 +0000 2024-04-28T12:42:41.704 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-28 12:34:58.774323428 +0000 2024-04-28T12:42:41.704 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T12:42:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-28T12:42:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-28T12:42:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000294804 s, 1.7 MB/s 2024-04-28T12:42:41.772 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T12:42:41.829 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-28 12:42:28.613006440 +0000 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-28 12:34:59.120315933 +0000 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-28 12:34:59.120315933 +0000 2024-04-28T12:42:41.887 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-28 12:34:59.120315933 +0000 2024-04-28T12:42:41.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T12:42:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-28T12:42:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-28T12:42:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000224006 s, 2.3 MB/s 2024-04-28T12:42:41.954 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T12:42:42.011 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-28 12:42:28.613006440 +0000 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-28 12:34:59.459308590 +0000 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-28 12:34:59.459308590 +0000 2024-04-28T12:42:42.068 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-28 12:34:59.459308590 +0000 2024-04-28T12:42:42.069 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T12:42:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-28T12:42:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-28T12:42:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00019367 s, 2.6 MB/s 2024-04-28T12:42:42.134 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T12:42:42.190 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-04-28T12:42:42.246 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-28 12:42:28.613006440 +0000 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-28 12:34:59.788301463 +0000 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-28 12:34:59.788301463 +0000 2024-04-28T12:42:42.247 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-28 12:34:59.788301463 +0000 2024-04-28T12:42:42.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T12:42:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-28T12:42:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-28T12:42:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000223945 s, 2.3 MB/s 2024-04-28T12:42:42.311 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T12:42:42.368 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-28T12:42:42.368 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T12:42:42.385 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T12:42:42.385 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-28 12:42:18.966956141 +0000 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-28 12:34:53.722839855 +0000 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-28 12:34:53.722839855 +0000 2024-04-28T12:42:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-28 12:34:53.722839855 +0000 2024-04-28T12:42:42.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T12:42:42.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:42 smithi077 ceph-mon[32553]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:42.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:42 smithi077 ceph-mon[32553]: Reconfiguring mon.c (monmap changed)... 2024-04-28T12:42:42.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:42 smithi077 ceph-mon[32553]: Reconfiguring daemon mon.c on smithi191 2024-04-28T12:42:42.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:42 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:42.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:42 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:42.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:42 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:42.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:42 smithi191 ceph-mon[33016]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:42.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:42 smithi191 ceph-mon[33016]: Reconfiguring mon.c (monmap changed)... 2024-04-28T12:42:42.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:42 smithi191 ceph-mon[33016]: Reconfiguring daemon mon.c on smithi191 2024-04-28T12:42:42.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:42 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:42.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:42 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:42.504 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:42 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:42.506 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-28T12:42:42.506 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-28T12:42:42.506 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000279738 s, 1.8 MB/s 2024-04-28T12:42:42.507 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T12:42:42.564 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-28 12:42:18.966956141 +0000 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-28 12:34:54.053833551 +0000 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-28 12:34:54.053833551 +0000 2024-04-28T12:42:42.620 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-28 12:34:54.053833551 +0000 2024-04-28T12:42:42.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T12:42:42.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:42 smithi053 ceph-mon[29182]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:42.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:42 smithi053 ceph-mon[29182]: Reconfiguring mon.c (monmap changed)... 2024-04-28T12:42:42.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:42 smithi053 ceph-mon[29182]: Reconfiguring daemon mon.c on smithi191 2024-04-28T12:42:42.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:42 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:42.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:42 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:42.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:42 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:42.683 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-28T12:42:42.683 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-28T12:42:42.683 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000204436 s, 2.5 MB/s 2024-04-28T12:42:42.685 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T12:42:42.740 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-04-28T12:42:42.795 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T12:42:42.795 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:42.795 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-28T12:42:42.795 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:42.795 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:42.795 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-28 12:42:18.966956141 +0000 2024-04-28T12:42:42.795 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-28 12:34:54.385827225 +0000 2024-04-28T12:42:42.796 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-28 12:34:54.385827225 +0000 2024-04-28T12:42:42.796 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-28 12:34:54.385827225 +0000 2024-04-28T12:42:42.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T12:42:42.858 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-28T12:42:42.865 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-28T12:42:42.865 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000180751 s, 2.8 MB/s 2024-04-28T12:42:42.866 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T12:42:42.921 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-04-28T12:42:42.976 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T12:42:42.976 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:42:42.976 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-28T12:42:42.976 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:42:42.976 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:42:42.977 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-28 12:42:18.967956122 +0000 2024-04-28T12:42:42.977 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-28 12:34:54.722820804 +0000 2024-04-28T12:42:42.977 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-28 12:34:54.722820804 +0000 2024-04-28T12:42:42.977 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-28 12:34:54.722820804 +0000 2024-04-28T12:42:42.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T12:42:43.038 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-28T12:42:43.039 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-28T12:42:43.039 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00019243 s, 2.7 MB/s 2024-04-28T12:42:43.040 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T12:42:43.097 INFO:tasks.cephadm:Deploying osd.0 on smithi053 with /dev/vg_nvme/lv_4... 2024-04-28T12:42:43.097 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T12:42:43.342 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:42:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:42:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:43.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:42:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:44.492 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:44 smithi053 ceph-mon[29182]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:44.492 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:44 smithi053 ceph-mon[29182]: Deploying daemon mgr.z on smithi191 2024-04-28T12:42:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:44 smithi191 ceph-mon[33016]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:44 smithi191 ceph-mon[33016]: Deploying daemon mgr.z on smithi191 2024-04-28T12:42:44.538 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:44 smithi191 systemd[1]: Starting Ceph mgr.z for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:42:44.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:44 smithi077 ceph-mon[32553]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:44.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:44 smithi077 ceph-mon[32553]: Deploying daemon mgr.z on smithi191 2024-04-28T12:42:44.908 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:44 smithi191 podman[34855]: 2024-04-28 12:42:44.554616116 +0000 UTC m=+0.018494001 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:42:44.908 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:44 smithi191 podman[34855]: 2024-04-28 12:42:44.660528262 +0000 UTC m=+0.124406141 container create 0f9a78af91f4b6eacec2ad826ac6cab2bd90e503cc17ced5d122479171ca0ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-28T12:42:44.908 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:44 smithi191 podman[34855]: 2024-04-28 12:42:44.912945676 +0000 UTC m=+0.376823549 container init 0f9a78af91f4b6eacec2ad826ac6cab2bd90e503cc17ced5d122479171ca0ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-28T12:42:45.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:44 smithi191 podman[34855]: 2024-04-28 12:42:44.915643425 +0000 UTC m=+0.379521301 container start 0f9a78af91f4b6eacec2ad826ac6cab2bd90e503cc17ced5d122479171ca0ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-28T12:42:45.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:44 smithi191 ceph-mgr[34874]: -- 172.21.15.191:0/2544425807 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e0c5f7e5a0 con 0x55e0c5f52c00 2024-04-28T12:42:45.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:45 smithi191 bash[34855]: 0f9a78af91f4b6eacec2ad826ac6cab2bd90e503cc17ced5d122479171ca0ec5 2024-04-28T12:42:45.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:45 smithi191 systemd[1]: Started Ceph mgr.z for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:42:45.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:45.062+0000 7f9c7ca29200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:42:45.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:45.249+0000 7f9c7ca29200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:42:45.356 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T12:42:45.356 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T12:42:45.356 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-28T12:42:45.356 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-28T12:42:45.356 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00852937 s, 1.2 GB/s 2024-04-28T12:42:45.357 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-28T12:42:45.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:45.350+0000 7f9c7ca29200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:42:46.037 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:45.653+0000 7f9c7ca29200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:42:46.037 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:45.735+0000 7f9c7ca29200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:42:46.082 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_4 2024-04-28T12:42:46.339 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:46.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-mon[33016]: Deploying daemon mgr.y on smithi077 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:46.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:46 smithi053 ceph-mon[29182]: Deploying daemon mgr.y on smithi077 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:46.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:46 smithi077 ceph-mon[32553]: Deploying daemon mgr.y on smithi077 2024-04-28T12:42:47.198 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 systemd[1]: Starting Ceph mgr.y for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:42:47.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:46 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:46.952+0000 7f9c7ca29200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:42:47.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:47.123+0000 7f9c7ca29200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:42:47.586 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:47.303+0000 7f9c7ca29200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:42:47.586 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:47.590+0000 7f9c7ca29200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:42:47.632 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 podman[33920]: 2024-04-28 12:42:47.331625797 +0000 UTC m=+0.110304743 container create 371a85653376411011afc9a4cf808690ed4cfb8519b77bbd7a479a299f69ff73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-28T12:42:47.632 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 podman[33920]: 2024-04-28 12:42:47.239791501 +0000 UTC m=+0.018470462 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:42:47.632 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 podman[33920]: 2024-04-28 12:42:47.535042421 +0000 UTC m=+0.313721374 container init 371a85653376411011afc9a4cf808690ed4cfb8519b77bbd7a479a299f69ff73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-28T12:42:47.632 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 podman[33920]: 2024-04-28 12:42:47.538052332 +0000 UTC m=+0.316731278 container start 371a85653376411011afc9a4cf808690ed4cfb8519b77bbd7a479a299f69ff73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-28T12:42:47.632 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 ceph-mgr[33947]: -- 172.21.15.77:0/1058254767 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562763e8a5a0 con 0x562763e5e800 2024-04-28T12:42:47.632 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 bash[33920]: 371a85653376411011afc9a4cf808690ed4cfb8519b77bbd7a479a299f69ff73 2024-04-28T12:42:47.883 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:47.682+0000 7f9c7ca29200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:42:47.884 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:47.887+0000 7f9c7ca29200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:42:47.895 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 systemd[1]: Started Ceph mgr.y for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:42:47.895 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:47.706+0000 7ff752bc6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:42:47.895 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:47 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:47.899+0000 7ff752bc6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 -- 172.21.15.53:0/2239955706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 msgr2=0x7f3a7c1038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 --2- 172.21.15.53:0/2239955706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1038a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0099c0 tx=0x7f3a6c009cd0 comp rx=0 tx=0).stop 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 -- 172.21.15.53:0/2239955706 shutdown_connections 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 --2- 172.21.15.53:0/2239955706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1038a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 --2- 172.21.15.53:0/2239955706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c105020 0x7f3a7c10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 --2- 172.21.15.53:0/2239955706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c104680 0x7f3a7c104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 -- 172.21.15.53:0/2239955706 >> 172.21.15.53:0/2239955706 conn(0x7f3a7c0fea20 msgr2=0x7f3a7c100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 -- 172.21.15.53:0/2239955706 shutdown_connections 2024-04-28T12:42:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.027+0000 7f3a80980700 1 -- 172.21.15.53:0/2239955706 wait complete. 2024-04-28T12:42:48.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 Processor -- start 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 -- start start 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c104680 0x7f3a7c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c105020 0x7f3a7c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c10ef20 con 0x7f3a7c103480 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c10eda0 con 0x7f3a7c104680 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a80980700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c10f0a0 con 0x7f3a7c105020 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a7a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a7a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49648/0 (socket says 172.21.15.53:49648) 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.028+0000 7f3a7a59c700 1 -- 172.21.15.53:0/3485526118 learned_addr learned my addr 172.21.15.53:0/3485526118 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a79d9b700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c104680 0x7f3a7c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a7ad9d700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c105020 0x7f3a7c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a7a59c700 1 -- 172.21.15.53:0/3485526118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c105020 msgr2=0x7f3a7c1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a7a59c700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c105020 0x7f3a7c1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a7a59c700 1 -- 172.21.15.53:0/3485526118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c104680 msgr2=0x7f3a7c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a7a59c700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c104680 0x7f3a7c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a7a59c700 1 -- 172.21.15.53:0/3485526118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c1a1250 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a7a59c700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1995f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00f710 tx=0x7f3a6c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c009e60 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c005940 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0054f0 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7c1a14e0 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.029+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a7c1a1970 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.030+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3a6c02b440 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.030+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a7c06a110 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.030+0000 7f3a6b7fe700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3a64039890 0x7f3a6403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.030+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3a6c04d980 con 0x7f3a7c103480 2024-04-28T12:42:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.031+0000 7f3a79d9b700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3a64039890 0x7f3a6403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.031+0000 7f3a79d9b700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3a64039890 0x7f3a6403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009c60 tx=0x7f3a70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:48.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.036+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a6c013070 con 0x7f3a7c103480 2024-04-28T12:42:48.198 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:48.009+0000 7ff752bc6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:42:48.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-mon[32553]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:48.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:48.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:48.255+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3a7c063860 con 0x7f3a64039890 2024-04-28T12:42:48.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:48.086+0000 7f9c7ca29200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:42:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-mon[33016]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:48 smithi053 ceph-mon[29182]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:48.698 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:48.337+0000 7ff752bc6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:42:48.698 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:48.431+0000 7ff752bc6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:42:48.918 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:48.713+0000 7f9c7ca29200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:42:49.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:49.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:49.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:49.244 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:49.244 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:49.244 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:49.245 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:49.149+0000 7f9c7ca29200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:42:49.245 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:49.248+0000 7f9c7ca29200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:42:49.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:49.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:49.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:49.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:49.452+0000 7f9c7ca29200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:42:49.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:49.676+0000 7ff752bc6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:42:49.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:49.842+0000 7ff752bc6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:42:50.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:49.970+0000 7f9c7ca29200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:42:50.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:50.061+0000 7f9c7ca29200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:50.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:50.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:50.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.290 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:50.011+0000 7ff752bc6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:42:50.290 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:50.294+0000 7ff752bc6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:50.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:50.582 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:50.385+0000 7ff752bc6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:42:50.582 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:50.585+0000 7ff752bc6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:42:50.639 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:50.301+0000 7f9c7ca29200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:42:50.640 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:50.433+0000 7f9c7ca29200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:42:50.640 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:50.528+0000 7f9c7ca29200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:42:50.640 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:50.644+0000 7f9c7ca29200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:42:50.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:50.775+0000 7ff752bc6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:42:51.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:50.963+0000 7f9c7ca29200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:42:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-mon[33016]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T12:42:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-mon[33016]: Reconfiguring daemon mgr.x on smithi053 2024-04-28T12:42:51.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3133913705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa"}]: dispatch 2024-04-28T12:42:51.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3133913705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa"}]': finished 2024-04-28T12:42:51.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-mon[33016]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T12:42:51.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:51.448 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:51.372+0000 7ff752bc6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:42:51.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-mon[32553]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T12:42:51.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-mon[32553]: Reconfiguring daemon mgr.x on smithi053 2024-04-28T12:42:51.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3133913705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa"}]: dispatch 2024-04-28T12:42:51.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3133913705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa"}]': finished 2024-04-28T12:42:51.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-mon[32553]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T12:42:51.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:51.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:51 smithi053 ceph-mon[29182]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T12:42:51.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:51 smithi053 ceph-mon[29182]: Reconfiguring daemon mgr.x on smithi053 2024-04-28T12:42:51.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:51 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3133913705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa"}]: dispatch 2024-04-28T12:42:51.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:51 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3133913705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa"}]': finished 2024-04-28T12:42:51.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:51 smithi053 ceph-mon[29182]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T12:42:51.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:52.037 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:51.561+0000 7f9c7ca29200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:42:52.037 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 12:42:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z[34870]: 2024-04-28T12:42:51.778+0000 7f9c7ca29200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:42:52.074 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:51.787+0000 7ff752bc6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:42:52.074 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:51.880+0000 7ff752bc6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:42:52.074 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:52.077+0000 7ff752bc6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:42:52.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:52.169+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f3a6c00fb70 con 0x7f3a7c103480 2024-04-28T12:42:52.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-mon[32553]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:52.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3123146244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:52.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-mon[32553]: Standby manager daemon z started 2024-04-28T12:42:52.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T12:42:52.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:52.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T12:42:52.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:52 smithi191 ceph-mon[33016]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:52.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:52 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3123146244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:52.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:52 smithi191 ceph-mon[33016]: Standby manager daemon z started 2024-04-28T12:42:52.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:52 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T12:42:52.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:52 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:52.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:52 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T12:42:52.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:52 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:52.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:52 smithi053 ceph-mon[29182]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:52.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:52 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3123146244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:52.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:52 smithi053 ceph-mon[29182]: Standby manager daemon z started 2024-04-28T12:42:52.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:52 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T12:42:52.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:52 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:52.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:52 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T12:42:52.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:52 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.191:0/2503881105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:52.912 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:52.590+0000 7ff752bc6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:42:52.912 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:52.680+0000 7ff752bc6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:42:53.166 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:52.916+0000 7ff752bc6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:42:53.166 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:53.047+0000 7ff752bc6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:42:53.166 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:53.140+0000 7ff752bc6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:42:53.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:53 smithi077 ceph-mon[32553]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-28T12:42:53.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:53 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T12:42:53.448 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:53.258+0000 7ff752bc6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:42:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:53 smithi191 ceph-mon[33016]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-28T12:42:53.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:53 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T12:42:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:53 smithi053 ceph-mon[29182]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-28T12:42:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:53 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T12:42:53.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:53.557+0000 7ff752bc6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:42:54.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:54 smithi077 ceph-mon[32553]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:54.448 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:54.148+0000 7ff752bc6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:42:54.448 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 12:42:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y[33935]: 2024-04-28T12:42:54.362+0000 7ff752bc6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:42:54.488 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:54 smithi053 ceph-mon[29182]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:54 smithi191 ceph-mon[33016]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:55.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:42:55.184+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f3a6c02bca0 con 0x7f3a7c103480 2024-04-28T12:42:55.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:55 smithi053 ceph-mon[29182]: Standby manager daemon y started 2024-04-28T12:42:55.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:55 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T12:42:55.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:55 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:55.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:55 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T12:42:55.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:55 smithi053 ceph-mon[29182]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:55.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:55 smithi077 ceph-mon[32553]: Standby manager daemon y started 2024-04-28T12:42:55.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:55 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T12:42:55.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:55 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:55.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:55 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T12:42:55.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:55 smithi077 ceph-mon[32553]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:55 smithi191 ceph-mon[33016]: Standby manager daemon y started 2024-04-28T12:42:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:55 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T12:42:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:55 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:55 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T12:42:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:55 smithi191 ceph-mon[33016]: from='mgr.? 172.21.15.77:0/4216537255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:56.373 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:56 smithi053 ceph-mon[29182]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:56.373 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:56 smithi053 ceph-mon[29182]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-28T12:42:56.373 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:56 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T12:42:56.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:56 smithi077 ceph-mon[32553]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:56.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:56 smithi077 ceph-mon[32553]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-28T12:42:56.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:56 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T12:42:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:56 smithi191 ceph-mon[33016]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:56 smithi191 ceph-mon[33016]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-28T12:42:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:56 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T12:42:57.559 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.559 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.559 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:57.560 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:57.560 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:57.560 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.560 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:57 smithi053 ceph-mon[29182]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:57.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:57.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:57.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:57.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:57 smithi077 ceph-mon[32553]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:42:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:57 smithi191 ceph-mon[33016]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:59.528 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:59 smithi053 ceph-mon[29182]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:59.528 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T12:42:59.528 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:59.528 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:42:59 smithi053 ceph-mon[29182]: Deploying daemon osd.0 on smithi053 2024-04-28T12:42:59.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:59 smithi077 ceph-mon[32553]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:59.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T12:42:59.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:59.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:42:59 smithi077 ceph-mon[32553]: Deploying daemon osd.0 on smithi053 2024-04-28T12:42:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:59 smithi191 ceph-mon[33016]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T12:42:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:42:59 smithi191 ceph-mon[33016]: Deploying daemon osd.0 on smithi053 2024-04-28T12:43:01.512 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:01 smithi053 ceph-mon[29182]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:01.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:01 smithi077 ceph-mon[32553]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:01 smithi191 ceph-mon[33016]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:03.522 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:03 smithi053 ceph-mon[29182]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:03.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:03 smithi077 ceph-mon[32553]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:03.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:03 smithi191 ceph-mon[33016]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:04.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:04.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:04.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:04.782 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:04.782 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:04.782 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:06.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:06 smithi053 ceph-mon[29182]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:06.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:06 smithi077 ceph-mon[32553]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:06 smithi191 ceph-mon[33016]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:07.166 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 0 on host 'smithi053' 2024-04-28T12:43:07.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.169+0000 7f3a6b7fe700 1 -- 172.21.15.53:0/3485526118 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3a7c063860 con 0x7f3a64039890 2024-04-28T12:43:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3a64039890 msgr2=0x7f3a6403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3a64039890 0x7f3a6403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009c60 tx=0x7f3a70009260 comp rx=0 tx=0).stop 2024-04-28T12:43:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 msgr2=0x7f3a7c1995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1995f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00f710 tx=0x7f3a6c00f740 comp rx=0 tx=0).stop 2024-04-28T12:43:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 shutdown_connections 2024-04-28T12:43:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3a64039890 0x7f3a6403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a7c103480 0x7f3a7c1995f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c105020 0x7f3a7c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 --2- 172.21.15.53:0/3485526118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c104680 0x7f3a7c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 >> 172.21.15.53:0/3485526118 conn(0x7f3a7c0fea20 msgr2=0x7f3a7c107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 shutdown_connections 2024-04-28T12:43:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:43:07.172+0000 7f3a80980700 1 -- 172.21.15.53:0/3485526118 wait complete. 2024-04-28T12:43:07.796 DEBUG:teuthology.orchestra.run.smithi053:osd.0> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.0.service 2024-04-28T12:43:07.798 INFO:tasks.cephadm:Deploying osd.1 on smithi077 with /dev/vg_nvme/lv_4... 2024-04-28T12:43:07.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T12:43:08.040 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:43:08.145 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:43:07 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0[43553]: 2024-04-28T12:43:07.979+0000 7f72ea5847c0 -1 osd.0 0 log_to_monitors true 2024-04-28T12:43:08.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:08 smithi077 ceph-mon[32553]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:08.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:08.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:08.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:08 smithi077 ceph-mon[32553]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T12:43:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:08 smithi191 ceph-mon[33016]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:08 smithi191 ceph-mon[33016]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T12:43:08.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:08 smithi053 ceph-mon[29182]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:08.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:08.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:08.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:08 smithi053 ceph-mon[29182]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T12:43:09.260 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:09 smithi077 ceph-mon[32553]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T12:43:09.260 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:09 smithi077 ceph-mon[32553]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T12:43:09.260 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:09 smithi077 ceph-mon[32553]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:43:09.260 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:09.261 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:09.261 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:09.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-mon[29182]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T12:43:09.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-mon[29182]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T12:43:09.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-mon[29182]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:43:09.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:09.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:09.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:09 smithi191 ceph-mon[33016]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T12:43:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:09 smithi191 ceph-mon[33016]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T12:43:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:09 smithi191 ceph-mon[33016]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:43:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:09.859 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T12:43:09.859 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T12:43:09.859 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-28T12:43:09.859 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-28T12:43:09.859 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112175 s, 935 MB/s 2024-04-28T12:43:09.859 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-28T12:43:09.873 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0[43553]: 2024-04-28T12:43:09.608+0000 7f72dfbe9700 -1 osd.0 0 waiting for initial osdmap 2024-04-28T12:43:09.873 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:43:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0[43553]: 2024-04-28T12:43:09.611+0000 7f72d79d4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:43:10.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:10 smithi053 ceph-mon[29182]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:10.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:10 smithi053 ceph-mon[29182]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:43:10.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:10 smithi053 ceph-mon[29182]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T12:43:10.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:10.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:10.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:10 smithi053 ceph-mon[29182]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' 2024-04-28T12:43:10.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:10 smithi077 ceph-mon[32553]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:10.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:10 smithi077 ceph-mon[32553]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:43:10.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:10 smithi077 ceph-mon[32553]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T12:43:10.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:10.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:10.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:10 smithi077 ceph-mon[32553]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' 2024-04-28T12:43:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:10 smithi191 ceph-mon[33016]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:10 smithi191 ceph-mon[33016]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:43:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:10 smithi191 ceph-mon[33016]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T12:43:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:10 smithi191 ceph-mon[33016]: from='osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705]' entity='osd.0' 2024-04-28T12:43:10.582 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-04-28T12:43:10.821 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:43:11.283 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:11 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:43:11.284 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:11 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:43:11.284 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:11.284 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:11 smithi053 ceph-mon[29182]: osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] boot 2024-04-28T12:43:11.284 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:11 smithi053 ceph-mon[29182]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T12:43:11.284 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:11.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:11 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:43:11.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:11 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:43:11.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:11.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:11 smithi077 ceph-mon[32553]: osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] boot 2024-04-28T12:43:11.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:11 smithi077 ceph-mon[32553]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T12:43:11.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:11 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:43:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:11 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:43:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:11 smithi191 ceph-mon[33016]: osd.0 [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] boot 2024-04-28T12:43:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:11 smithi191 ceph-mon[33016]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T12:43:11.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:43:12.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/1171004053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 msgr2=0x7f7d58108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:12.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/1171004053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58108c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009990 tx=0x7f7d4c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/1171004053 shutdown_connections 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/1171004053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d58109190 0x7f7d5810b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/1171004053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d58103f30 0x7f7d58106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/1171004053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58108c50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/1171004053 >> 172.21.15.77:0/1171004053 conn(0x7f7d580fd940 msgr2=0x7f7d580ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/1171004053 shutdown_connections 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/1171004053 wait complete. 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 Processor -- start 2024-04-28T12:43:12.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.403+0000 7f7d5e4c0700 1 -- start start 2024-04-28T12:43:12.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d5e4c0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d58103f30 0x7f7d58199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d5e4c0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d5e4c0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d58109190 0x7f7d581a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d5e4c0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810ef60 con 0x7f7d58103f30 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d5e4c0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810ede0 con 0x7f7d58106860 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d5e4c0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810f0e0 con 0x7f7d58109190 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38254/0 (socket says 172.21.15.77:38254) 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 -- 172.21.15.77:0/4205431978 learned_addr learned my addr 172.21.15.77:0/4205431978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d57fff700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d58103f30 0x7f7d58199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d5ca5d700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d58109190 0x7f7d581a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 -- 172.21.15.77:0/4205431978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d58109190 msgr2=0x7f7d581a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d58109190 0x7f7d581a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 -- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d58103f30 msgr2=0x7f7d58199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d58103f30 0x7f7d58199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 -- 172.21.15.77:0/4205431978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d581a1270 con 0x7f7d58106860 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d577fe700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58199b50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c006010 tx=0x7f7d4c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.404+0000 7f7d557fa700 1 -- 172.21.15.77:0/4205431978 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c027070 con 0x7f7d58106860 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.405+0000 7f7d557fa700 1 -- 172.21.15.77:0/4205431978 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d4c004000 con 0x7f7d58106860 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.405+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d581a1500 con 0x7f7d58106860 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.405+0000 7f7d557fa700 1 -- 172.21.15.77:0/4205431978 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c022c80 con 0x7f7d58106860 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.405+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d581a1950 con 0x7f7d58106860 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.406+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d5806a110 con 0x7f7d58106860 2024-04-28T12:43:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.407+0000 7f7d557fa700 1 -- 172.21.15.77:0/4205431978 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f7d4c02a070 con 0x7f7d58106860 2024-04-28T12:43:12.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.407+0000 7f7d557fa700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d400a3bb0 0x7f7d400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:12.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.407+0000 7f7d557fa700 1 -- 172.21.15.77:0/4205431978 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f7d4c0cf8c0 con 0x7f7d58106860 2024-04-28T12:43:12.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.407+0000 7f7d57fff700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d400a3bb0 0x7f7d400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:12.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.408+0000 7f7d57fff700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d400a3bb0 0x7f7d400a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d48004630 tx=0x7f7d4800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:12.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.412+0000 7f7d557fa700 1 -- 172.21.15.77:0/4205431978 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7d4c05f060 con 0x7f7d58106860 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T12:43:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:12.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:12.623+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7d58063860 con 0x7f7d400a3bb0 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:12.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:12.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:13.432 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi053 2024-04-28T12:43:13.433 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi053 to 17093M 2024-04-28T12:43:13.433 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:13.433 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:13.433 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:13.433 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:13.433 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:13.433 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi053 2024-04-28T12:43:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi053 to 17093M 2024-04-28T12:43:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi053 2024-04-28T12:43:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi053 to 17093M 2024-04-28T12:43:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:14.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:14 smithi077 ceph-mon[32553]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:14.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:14 smithi077 ceph-mon[32553]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:14.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:14 smithi191 ceph-mon[33016]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:14.438 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:14 smithi191 ceph-mon[33016]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:14.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:14 smithi053 ceph-mon[29182]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:14.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:14 smithi053 ceph-mon[29182]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:16.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:16 smithi077 ceph-mon[32553]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:16.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:16 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/3298557749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]: dispatch 2024-04-28T12:43:16.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:16 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]: dispatch 2024-04-28T12:43:16.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:16 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]': finished 2024-04-28T12:43:16.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:16 smithi077 ceph-mon[32553]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T12:43:16.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:16.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:16 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/3790688689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:16 smithi191 ceph-mon[33016]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:16.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:16 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/3298557749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]: dispatch 2024-04-28T12:43:16.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:16 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]: dispatch 2024-04-28T12:43:16.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:16 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]': finished 2024-04-28T12:43:16.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:16 smithi191 ceph-mon[33016]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T12:43:16.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:16.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:16 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/3790688689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:16.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:16 smithi053 ceph-mon[29182]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:16.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:16 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/3298557749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]: dispatch 2024-04-28T12:43:16.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:16 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]: dispatch 2024-04-28T12:43:16.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:16 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f15997c2-61f7-4970-b488-c38990780d97"}]': finished 2024-04-28T12:43:16.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:16 smithi053 ceph-mon[29182]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T12:43:16.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:16.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:16 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/3790688689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:18.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:18 smithi077 ceph-mon[32553]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:18 smithi191 ceph-mon[33016]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:18.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:18 smithi053 ceph-mon[29182]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:20.388 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:20 smithi077 ceph-mon[32553]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:20.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:20 smithi191 ceph-mon[33016]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:20 smithi053 ceph-mon[29182]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:22 smithi077 ceph-mon[32553]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:22 smithi191 ceph-mon[33016]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:22.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:22 smithi053 ceph-mon[29182]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:23.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:23 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T12:43:23.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:23 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:23 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T12:43:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:23 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:23.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:23 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T12:43:23.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:23 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:24.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:24 smithi077 ceph-mon[32553]: Deploying daemon osd.1 on smithi077 2024-04-28T12:43:24.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:24 smithi077 ceph-mon[32553]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:24 smithi191 ceph-mon[33016]: Deploying daemon osd.1 on smithi077 2024-04-28T12:43:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:24 smithi191 ceph-mon[33016]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:24.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:24 smithi053 ceph-mon[29182]: Deploying daemon osd.1 on smithi077 2024-04-28T12:43:24.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:24 smithi053 ceph-mon[29182]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:26.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:26 smithi077 ceph-mon[32553]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:26 smithi191 ceph-mon[33016]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:26 smithi053 ceph-mon[29182]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:28.419 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:28 smithi077 ceph-mon[32553]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:28.419 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:28.419 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:28.419 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:28 smithi191 ceph-mon[33016]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:28 smithi053 ceph-mon[29182]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:30.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:30 smithi077 ceph-mon[32553]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:30.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:30.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:30.374 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:30 smithi053 ceph-mon[29182]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:30.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:30.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:30.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:30 smithi191 ceph-mon[33016]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.440+0000 7f7d557fa700 1 -- 172.21.15.77:0/4205431978 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7d58063860 con 0x7f7d400a3bb0 2024-04-28T12:43:31.437 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 1 on host 'smithi077' 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.442+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d400a3bb0 msgr2=0x7f7d400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.442+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d400a3bb0 0x7f7d400a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d48004630 tx=0x7f7d4800b3c0 comp rx=0 tx=0).stop 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.442+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 msgr2=0x7f7d58199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.442+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58199b50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c006010 tx=0x7f7d4c0047b0 comp rx=0 tx=0).stop 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 shutdown_connections 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d400a3bb0 0x7f7d400a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d58103f30 0x7f7d58199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d58109190 0x7f7d581a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 --2- 172.21.15.77:0/4205431978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d58106860 0x7f7d58199b50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 >> 172.21.15.77:0/4205431978 conn(0x7f7d580fd940 msgr2=0x7f7d58105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 shutdown_connections 2024-04-28T12:43:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:43:31.443+0000 7f7d5e4c0700 1 -- 172.21.15.77:0/4205431978 wait complete. 2024-04-28T12:43:32.090 DEBUG:teuthology.orchestra.run.smithi077:osd.1> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.1.service 2024-04-28T12:43:32.092 INFO:tasks.cephadm:Deploying osd.2 on smithi191 with /dev/vg_nvme/lv_4... 2024-04-28T12:43:32.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T12:43:32.342 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:43:32.432 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:32 smithi077 ceph-mon[32553]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:32.432 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:32.432 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:32.432 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:32.432 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:32 smithi077 ceph-mon[32553]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:43:32.458 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:32 smithi191 ceph-mon[33016]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:32.458 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:32.458 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:32.458 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:32.458 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:32 smithi191 ceph-mon[33016]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:43:32.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:32 smithi053 ceph-mon[29182]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:32.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:32.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:32.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:32.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:32 smithi053 ceph-mon[29182]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:43:33.604 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:33 smithi191 ceph-mon[33016]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:33.604 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:33 smithi191 ceph-mon[33016]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T12:43:33.604 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:33 smithi191 ceph-mon[33016]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T12:43:33.604 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:33 smithi191 ceph-mon[33016]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:43:33.604 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:33.630 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:33 smithi077 ceph-mon[32553]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:33.630 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:33 smithi077 ceph-mon[32553]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T12:43:33.631 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:33 smithi077 ceph-mon[32553]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T12:43:33.631 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:33 smithi077 ceph-mon[32553]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:43:33.631 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:33.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:43:33 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1[37141]: 2024-04-28T12:43:33.442+0000 7fdca9777700 -1 osd.1 0 waiting for initial osdmap 2024-04-28T12:43:33.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:43:33 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1[37141]: 2024-04-28T12:43:33.448+0000 7fdca556a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:43:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:33 smithi053 ceph-mon[29182]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:33 smithi053 ceph-mon[29182]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T12:43:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:33 smithi053 ceph-mon[29182]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T12:43:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:33 smithi053 ceph-mon[29182]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:43:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:34.174 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T12:43:34.174 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T12:43:34.174 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-28T12:43:34.174 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-28T12:43:34.174 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112102 s, 935 MB/s 2024-04-28T12:43:34.174 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-28T12:43:34.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:34 smithi077 ceph-mon[32553]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:43:34.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:34 smithi077 ceph-mon[32553]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T12:43:34.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:34.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:34.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:34 smithi191 ceph-mon[33016]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:43:34.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:34 smithi191 ceph-mon[33016]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T12:43:34.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:34.785 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:34.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2024-04-28T12:43:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:34 smithi053 ceph-mon[29182]: from='osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:43:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:34 smithi053 ceph-mon[29182]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T12:43:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:35.051 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:43:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] boot 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi077 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi077 to 17088M 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:35.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:43:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:43:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] boot 2024-04-28T12:43:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi077 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi077 to 17088M 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: osd.1 [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] boot 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi077 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi077 to 17088M 2024-04-28T12:43:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:36.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:36 smithi191 ceph-mon[33016]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 -- 172.21.15.191:0/3781343238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9450105040 msgr2=0x7f945010b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 --2- 172.21.15.191:0/3781343238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9450105040 0x7f945010b8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f944800b0a0 tx=0x7f944800b3b0 comp rx=0 tx=0).stop 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 -- 172.21.15.191:0/3781343238 shutdown_connections 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 --2- 172.21.15.191:0/3781343238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94501034a0 0x7f94501038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 --2- 172.21.15.191:0/3781343238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9450105040 0x7f945010b8f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 --2- 172.21.15.191:0/3781343238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94501046a0 0x7f9450104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 -- 172.21.15.191:0/3781343238 >> 172.21.15.191:0/3781343238 conn(0x7f94500fea40 msgr2=0x7f9450100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 -- 172.21.15.191:0/3781343238 shutdown_connections 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 -- 172.21.15.191:0/3781343238 wait complete. 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 Processor -- start 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.585+0000 7f94586de700 1 -- start start 2024-04-28T12:43:36.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94501034a0 0x7f9450199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94501046a0 0x7f9450199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9450105040 0x7f94501a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f945010ef40 con 0x7f9450105040 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f945010edc0 con 0x7f94501046a0 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f945010f0c0 con 0x7f94501034a0 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94501034a0 0x7f9450199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94501034a0 0x7f9450199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:34742/0 (socket says 172.21.15.191:34742) 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 -- 172.21.15.191:0/1313693805 learned_addr learned my addr 172.21.15.191:0/1313693805 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f9456c7b700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9450105040 0x7f94501a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f9455c79700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94501046a0 0x7f9450199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 -- 172.21.15.191:0/1313693805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94501046a0 msgr2=0x7f9450199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94501046a0 0x7f9450199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 -- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9450105040 msgr2=0x7f94501a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:36.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9450105040 0x7f94501a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 -- 172.21.15.191:0/1313693805 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94501a11b0 con 0x7f94501034a0 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f9456c7b700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9450105040 0x7f94501a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f9455c79700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94501046a0 0x7f9450199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f945647a700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94501034a0 0x7f9450199550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e9a0 tx=0x7f944c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94477fe700 1 -- 172.21.15.191:0/1313693805 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c00cbd0 con 0x7f94501034a0 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94501a14a0 con 0x7f94501034a0 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94477fe700 1 -- 172.21.15.191:0/1313693805 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f944c0044d0 con 0x7f94501034a0 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94477fe700 1 -- 172.21.15.191:0/1313693805 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c003e30 con 0x7f94501034a0 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.586+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94501a19e0 con 0x7f94501034a0 2024-04-28T12:43:36.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.587+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f945006a110 con 0x7f94501034a0 2024-04-28T12:43:36.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.588+0000 7f94477fe700 1 -- 172.21.15.191:0/1313693805 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f944c004000 con 0x7f94501034a0 2024-04-28T12:43:36.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.589+0000 7f94477fe700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f943c0a3bb0 0x7f943c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:36.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.589+0000 7f94477fe700 1 -- 172.21.15.191:0/1313693805 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f944c0cef70 con 0x7f94501034a0 2024-04-28T12:43:36.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.590+0000 7f9455c79700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f943c0a3bb0 0x7f943c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:36.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.590+0000 7f9455c79700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f943c0a3bb0 0x7f943c0a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9440009be0 tx=0x7f9440009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:36.590 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.593+0000 7f94477fe700 1 -- 172.21.15.191:0/1313693805 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f944c0042f0 con 0x7f94501034a0 2024-04-28T12:43:36.795 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:36.799+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9450063860 con 0x7f943c0a3bb0 2024-04-28T12:43:36.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:36 smithi053 ceph-mon[29182]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T12:43:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:36 smithi077 ceph-mon[32553]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T12:43:37.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:37 smithi191 ceph-mon[33016]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:37.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:37 smithi191 ceph-mon[33016]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:37.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:37.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:37.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:37 smithi053 ceph-mon[29182]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:37 smithi053 ceph-mon[29182]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:37 smithi077 ceph-mon[32553]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:37 smithi077 ceph-mon[32553]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:39 smithi191 ceph-mon[33016]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:39 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/3396412518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]: dispatch 2024-04-28T12:43:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:39 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]: dispatch 2024-04-28T12:43:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:39 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]': finished 2024-04-28T12:43:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:39 smithi191 ceph-mon[33016]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T12:43:39.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:39 smithi053 ceph-mon[29182]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:39 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/3396412518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]: dispatch 2024-04-28T12:43:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:39 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]: dispatch 2024-04-28T12:43:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:39 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]': finished 2024-04-28T12:43:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:39 smithi053 ceph-mon[29182]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T12:43:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:39.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:39 smithi077 ceph-mon[32553]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:39.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:39 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/3396412518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]: dispatch 2024-04-28T12:43:39.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:39 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]: dispatch 2024-04-28T12:43:39.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:39 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "129b78c4-4920-41b7-bf2e-b512e63ae231"}]': finished 2024-04-28T12:43:39.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:39 smithi077 ceph-mon[32553]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T12:43:39.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:40.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:40 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/4234103684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:40.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:40 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/4234103684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:40.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:40 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/4234103684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:41.781 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:41 smithi053 ceph-mon[29182]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:41 smithi191 ceph-mon[33016]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:41.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:41 smithi077 ceph-mon[32553]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:43 smithi191 ceph-mon[33016]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:43 smithi053 ceph-mon[29182]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:43.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:43 smithi077 ceph-mon[32553]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:45 smithi191 ceph-mon[33016]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:45 smithi053 ceph-mon[29182]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:45 smithi077 ceph-mon[32553]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:47.720 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:47 smithi191 ceph-mon[33016]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:47.720 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T12:43:47.720 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:47.720 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:47 smithi191 ceph-mon[33016]: Deploying daemon osd.2 on smithi191 2024-04-28T12:43:47.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:47 smithi077 ceph-mon[32553]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:47.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T12:43:47.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:47.750 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:47 smithi077 ceph-mon[32553]: Deploying daemon osd.2 on smithi191 2024-04-28T12:43:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:47 smithi053 ceph-mon[29182]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T12:43:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:47 smithi053 ceph-mon[29182]: Deploying daemon osd.2 on smithi191 2024-04-28T12:43:49.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:49 smithi191 ceph-mon[33016]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:49 smithi053 ceph-mon[29182]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:49 smithi077 ceph-mon[32553]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:51.611 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:51 smithi191 ceph-mon[33016]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:51 smithi053 ceph-mon[29182]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:51.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:51 smithi077 ceph-mon[32553]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:53.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:53.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:53.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:53.180 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:53.180 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:53.180 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:53.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:53.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:53.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:53 smithi191 ceph-mon[33016]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:54.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:53 smithi053 ceph-mon[29182]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:54.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:53 smithi077 ceph-mon[32553]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:55.515 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.519+0000 7f94477fe700 1 -- 172.21.15.191:0/1313693805 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9450063860 con 0x7f943c0a3bb0 2024-04-28T12:43:55.516 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 2 on host 'smithi191' 2024-04-28T12:43:55.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f943c0a3bb0 msgr2=0x7f943c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:55.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f943c0a3bb0 0x7f943c0a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9440009be0 tx=0x7f9440009260 comp rx=0 tx=0).stop 2024-04-28T12:43:55.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94501034a0 msgr2=0x7f9450199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:55.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94501034a0 0x7f9450199550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e9a0 tx=0x7f944c00ed60 comp rx=0 tx=0).stop 2024-04-28T12:43:55.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 shutdown_connections 2024-04-28T12:43:55.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f943c0a3bb0 0x7f943c0a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:55.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9450105040 0x7f94501a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:55.519 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94501034a0 0x7f9450199550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:55.519 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 --2- 172.21.15.191:0/1313693805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94501046a0 0x7f9450199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:55.519 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 >> 172.21.15.191:0/1313693805 conn(0x7f94500fea40 msgr2=0x7f94501079f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:55.519 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 shutdown_connections 2024-04-28T12:43:55.519 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:43:55.522+0000 7f94586de700 1 -- 172.21.15.191:0/1313693805 wait complete. 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:55.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:56.162 DEBUG:teuthology.orchestra.run.smithi191:osd.2> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.2.service 2024-04-28T12:43:56.165 INFO:tasks.cephadm:Deploying osd.3 on smithi053 with /dev/vg_nvme/lv_3... 2024-04-28T12:43:56.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T12:43:56.424 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:43:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:56 smithi191 ceph-mon[33016]: from='osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:56 smithi191 ceph-mon[33016]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:56.834 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:56 smithi053 ceph-mon[29182]: from='osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:56.834 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:56 smithi053 ceph-mon[29182]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:56.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:56 smithi077 ceph-mon[32553]: from='osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:56.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:56 smithi077 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:57.806 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:57 smithi053 ceph-mon[29182]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:57.806 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:57 smithi053 ceph-mon[29182]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T12:43:57.806 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:57 smithi053 ceph-mon[29182]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T12:43:57.806 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:57.807 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:57 smithi053 ceph-mon[29182]: from='osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:43:57.807 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:57 smithi053 ceph-mon[29182]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:43:57.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:57 smithi077 ceph-mon[32553]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:57 smithi077 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T12:43:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:57 smithi077 ceph-mon[32553]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T12:43:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:57 smithi077 ceph-mon[32553]: from='osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:43:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:57 smithi077 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:43:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-mon[33016]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-mon[33016]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T12:43:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-mon[33016]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T12:43:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-mon[33016]: from='osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:43:58.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-mon[33016]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:43:58.038 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2[38184]: 2024-04-28T12:43:57.621+0000 7f534fe85700 -1 osd.2 0 waiting for initial osdmap 2024-04-28T12:43:58.038 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:43:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2[38184]: 2024-04-28T12:43:57.625+0000 7f534b477700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:43:58.395 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T12:43:58.396 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T12:43:58.396 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-28T12:43:58.396 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-28T12:43:58.396 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00846844 s, 1.2 GB/s 2024-04-28T12:43:58.396 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-28T12:43:58.735 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:58 smithi053 ceph-mon[29182]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:43:58.735 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:58 smithi053 ceph-mon[29182]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T12:43:58.735 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:58 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:58.735 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:58 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:58.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:58 smithi191 ceph-mon[33016]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:43:58.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:58 smithi191 ceph-mon[33016]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T12:43:58.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:58 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:58.743 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:58 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:58.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:58 smithi077 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:43:58.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:58 smithi077 ceph-mon[32553]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T12:43:58.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:58 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:58.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:58 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:59.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_3 2024-04-28T12:43:59.336 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] boot 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi191 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:59.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:59.732 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi191 to 17088M 2024-04-28T12:43:59.732 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:59.732 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:59.732 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:59.732 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:43:59 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:43:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:43:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] boot 2024-04-28T12:43:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T12:43:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi191 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi191 to 17088M 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:43:59 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:44:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:44:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:44:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: osd.2 [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] boot 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi191 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi191 to 17088M 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:43:59 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:00.764 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:00 smithi053 ceph-mon[29182]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T12:44:00.764 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:44:00.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:00 smithi077 ceph-mon[32553]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T12:44:00.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:44:01.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 -- 172.21.15.53:0/977191569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1034a0 msgr2=0x7f674c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:01.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 --2- 172.21.15.53:0/977191569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1034a0 0x7f674c1038c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f67400099c0 tx=0x7f6740009cd0 comp rx=0 tx=0).stop 2024-04-28T12:44:01.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 -- 172.21.15.53:0/977191569 shutdown_connections 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 --2- 172.21.15.53:0/977191569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1034a0 0x7f674c1038c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 --2- 172.21.15.53:0/977191569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f674c105040 0x7f674c10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 --2- 172.21.15.53:0/977191569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f674c1046a0 0x7f674c104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 -- 172.21.15.53:0/977191569 >> 172.21.15.53:0/977191569 conn(0x7f674c0fea40 msgr2=0x7f674c100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 -- 172.21.15.53:0/977191569 shutdown_connections 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 -- 172.21.15.53:0/977191569 wait complete. 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.022+0000 7f67512a4700 1 Processor -- start 2024-04-28T12:44:01.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f67512a4700 1 -- start start 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f67512a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f674c1034a0 0x7f674c199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f67512a4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1046a0 0x7f674c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f67512a4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f674c105040 0x7f674c1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f67512a4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f674c10ef50 con 0x7f674c1046a0 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f67512a4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f674c10edd0 con 0x7f674c1034a0 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f67512a4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f674c10f0d0 con 0x7f674c105040 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1046a0 0x7f674c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1046a0 0x7f674c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42354/0 (socket says 172.21.15.53:42354) 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 -- 172.21.15.53:0/2209452522 learned_addr learned my addr 172.21.15.53:0/2209452522 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:44:01.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674affd700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f674c1034a0 0x7f674c199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674b7fe700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f674c105040 0x7f674c1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 -- 172.21.15.53:0/2209452522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f674c105040 msgr2=0x7f674c1a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f674c105040 0x7f674c1a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 -- 172.21.15.53:0/2209452522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f674c1034a0 msgr2=0x7f674c199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f674c1034a0 0x7f674c199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 -- 172.21.15.53:0/2209452522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c1a1280 con 0x7f674c1046a0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.023+0000 7f674a7fc700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1046a0 0x7f674c199b60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f673c00d9b0 tx=0x7f673c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.024+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c017070 con 0x7f674c1046a0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.024+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c1a1570 con 0x7f674c1046a0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.024+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1a1ab0 con 0x7f674c1046a0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.024+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f673c005020 con 0x7f674c1046a0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.024+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c009b50 con 0x7f674c1046a0 2024-04-28T12:44:01.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.025+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c06a110 con 0x7f674c1046a0 2024-04-28T12:44:01.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.026+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f673c0051d0 con 0x7f674c1046a0 2024-04-28T12:44:01.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.027+0000 7f6733fff700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f67340a3c30 0x7f67340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:01.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.027+0000 7f674affd700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f67340a3c30 0x7f67340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:01.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.027+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f673c0cf200 con 0x7f674c1046a0 2024-04-28T12:44:01.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.027+0000 7f674affd700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f67340a3c30 0x7f67340a60f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67400038c0 tx=0x7f674000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:01.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.031+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f673c020020 con 0x7f674c1046a0 2024-04-28T12:44:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:00 smithi191 ceph-mon[33016]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T12:44:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:44:01.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:01.250+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f674c063860 con 0x7f67340a3c30 2024-04-28T12:44:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T12:44:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T12:44:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:44:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:01.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T12:44:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T12:44:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:44:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:02.510 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:44:02.510 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.510 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48139]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.511 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48139]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.511 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48143]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:44:02.511 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48143]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.511 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48143]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.511 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48143]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48146]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48146]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48146]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48146]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48149]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48149]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48149]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.511 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 sudo[48149]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41277]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41277]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41277]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41277]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41280]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41280]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41280]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.644 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41280]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.644 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41271]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:44:02.644 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41271]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.644 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41271]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41271]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41274]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41274]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41274]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 12:44:02 smithi077 sudo[41274]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42238]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42238]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42238]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42238]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.645 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42241]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:44:02.646 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42241]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.646 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42241]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.646 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42241]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42244]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42244]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42244]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42244]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42247]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42247]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42247]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:44:02.646 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 sudo[42247]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:44:02.647 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T12:44:02.647 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T12:44:02.647 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.647 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.647 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.844 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T12:44:02.844 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T12:44:02.844 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:02.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:02.846 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:02.846 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:02.846 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:02.846 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:02 smithi053 ceph-mon[29182]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:02.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:02 smithi077 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:44:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:44:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:44:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:44:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:02 smithi191 ceph-mon[33016]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:44:03.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:03.358+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f673c009cf0 con 0x7f674c1046a0 2024-04-28T12:44:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:03 smithi077 ceph-mon[32553]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:03 smithi077 ceph-mon[32553]: osdmap e22: 3 total, 3 up, 3 in 2024-04-28T12:44:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:03 smithi077 ceph-mon[32553]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-28T12:44:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:03 smithi191 ceph-mon[33016]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:03 smithi191 ceph-mon[33016]: osdmap e22: 3 total, 3 up, 3 in 2024-04-28T12:44:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:03 smithi191 ceph-mon[33016]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-28T12:44:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:03 smithi053 ceph-mon[29182]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:03 smithi053 ceph-mon[29182]: osdmap e22: 3 total, 3 up, 3 in 2024-04-28T12:44:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:03 smithi053 ceph-mon[29182]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-28T12:44:04.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:04 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/437042125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbade960-1344-45b7-a7ed-a056a7500a6f"}]: dispatch 2024-04-28T12:44:04.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:04 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/437042125' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbade960-1344-45b7-a7ed-a056a7500a6f"}]': finished 2024-04-28T12:44:04.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:04 smithi077 ceph-mon[32553]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T12:44:04.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:04.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:04 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3694703306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:04 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/437042125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbade960-1344-45b7-a7ed-a056a7500a6f"}]: dispatch 2024-04-28T12:44:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:04 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/437042125' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbade960-1344-45b7-a7ed-a056a7500a6f"}]': finished 2024-04-28T12:44:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:04 smithi191 ceph-mon[33016]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T12:44:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:04 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3694703306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:04 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/437042125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbade960-1344-45b7-a7ed-a056a7500a6f"}]: dispatch 2024-04-28T12:44:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:04 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/437042125' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbade960-1344-45b7-a7ed-a056a7500a6f"}]': finished 2024-04-28T12:44:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:04 smithi053 ceph-mon[29182]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T12:44:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:04 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3694703306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:05 smithi191 ceph-mon[33016]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:06.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:05 smithi053 ceph-mon[29182]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:06.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:05 smithi077 ceph-mon[32553]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:08.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:07 smithi191 ceph-mon[33016]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:08.320 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:07 smithi053 ceph-mon[29182]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:08.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:07 smithi077 ceph-mon[32553]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:10.236 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:09 smithi053 ceph-mon[29182]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:10.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:09 smithi191 ceph-mon[33016]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:10.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:09 smithi077 ceph-mon[32553]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:12.270 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:11 smithi053 ceph-mon[29182]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:12.270 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T12:44:12.270 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:11 smithi191 ceph-mon[33016]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T12:44:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:12.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:11 smithi077 ceph-mon[32553]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T12:44:12.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:13.236 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:12 smithi053 ceph-mon[29182]: Deploying daemon osd.3 on smithi053 2024-04-28T12:44:13.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:13 smithi191 ceph-mon[33016]: Deploying daemon osd.3 on smithi053 2024-04-28T12:44:13.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:12 smithi077 ceph-mon[32553]: Deploying daemon osd.3 on smithi053 2024-04-28T12:44:14.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:13 smithi053 ceph-mon[29182]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:14.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:13 smithi191 ceph-mon[33016]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:14.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:13 smithi077 ceph-mon[32553]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:16.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:16 smithi053 ceph-mon[29182]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:16 smithi191 ceph-mon[33016]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:16.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:16 smithi077 ceph-mon[32553]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:18.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:18 smithi053 ceph-mon[29182]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:18.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:18.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:18.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:18.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:18 smithi077 ceph-mon[32553]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:18.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:18.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:18.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:18 smithi191 ceph-mon[33016]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:20.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:20 smithi077 ceph-mon[32553]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:20.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:20 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:20 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:20 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:20.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:20 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:20.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:20 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:20 smithi191 ceph-mon[33016]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:20.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:20 smithi053 ceph-mon[29182]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:20 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:20 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:20 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:20 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:20 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:21.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.013+0000 7f6733fff700 1 -- 172.21.15.53:0/2209452522 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f674c063860 con 0x7f67340a3c30 2024-04-28T12:44:21.010 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 3 on host 'smithi053' 2024-04-28T12:44:21.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.015+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f67340a3c30 msgr2=0x7f67340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:21.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.015+0000 7f67512a4700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f67340a3c30 0x7f67340a60f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67400038c0 tx=0x7f674000b540 comp rx=0 tx=0).stop 2024-04-28T12:44:21.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1046a0 msgr2=0x7f674c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:21.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1046a0 0x7f674c199b60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f673c00d9b0 tx=0x7f673c00dcc0 comp rx=0 tx=0).stop 2024-04-28T12:44:21.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 shutdown_connections 2024-04-28T12:44:21.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f67340a3c30 0x7f67340a60f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:21.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f674c105040 0x7f674c1a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:21.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f674c1034a0 0x7f674c199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:21.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 --2- 172.21.15.53:0/2209452522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f674c1046a0 0x7f674c199b60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:21.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 >> 172.21.15.53:0/2209452522 conn(0x7f674c0fea40 msgr2=0x7f674c107a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:21.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 shutdown_connections 2024-04-28T12:44:21.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:44:21.016+0000 7f67512a4700 1 -- 172.21.15.53:0/2209452522 wait complete. 2024-04-28T12:44:21.733 DEBUG:teuthology.orchestra.run.smithi053:osd.3> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.3.service 2024-04-28T12:44:21.735 INFO:tasks.cephadm:Deploying osd.4 on smithi077 with /dev/vg_nvme/lv_3... 2024-04-28T12:44:21.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T12:44:21.982 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:44:22.104 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:22 smithi077 ceph-mon[32553]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:22.104 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:22.104 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:22.104 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:22.105 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:22 smithi077 ceph-mon[32553]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T12:44:22.233 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:22 smithi053 ceph-mon[29182]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:22.234 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:22.234 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:22.234 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:22.234 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:22 smithi053 ceph-mon[29182]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T12:44:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:22 smithi191 ceph-mon[33016]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:22 smithi191 ceph-mon[33016]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T12:44:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:23 smithi191 ceph-mon[33016]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T12:44:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:23 smithi191 ceph-mon[33016]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T12:44:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:23 smithi191 ceph-mon[33016]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:44:23.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:23 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:23.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:23 smithi053 ceph-mon[29182]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T12:44:23.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:23 smithi053 ceph-mon[29182]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T12:44:23.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:23 smithi053 ceph-mon[29182]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:44:23.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:23 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:23.325 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 12:44:23 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T12:44:23.015+0000 7f98402ee700 -1 osd.3 0 waiting for initial osdmap 2024-04-28T12:44:23.325 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 12:44:23 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T12:44:23.023+0000 7f983c0e1700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:44:23.376 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:23 smithi077 ceph-mon[32553]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T12:44:23.376 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:23 smithi077 ceph-mon[32553]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T12:44:23.376 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:23 smithi077 ceph-mon[32553]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:44:23.376 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:23 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T12:44:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T12:44:24.015 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-28T12:44:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-28T12:44:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117742 s, 891 MB/s 2024-04-28T12:44:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-28T12:44:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:24 smithi191 ceph-mon[33016]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:24 smithi191 ceph-mon[33016]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:44:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:24 smithi191 ceph-mon[33016]: osdmap e25: 4 total, 3 up, 4 in 2024-04-28T12:44:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:24 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:24 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:24.387 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:24 smithi077 ceph-mon[32553]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:24.387 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:24 smithi077 ceph-mon[32553]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:44:24.387 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:24 smithi077 ceph-mon[32553]: osdmap e25: 4 total, 3 up, 4 in 2024-04-28T12:44:24.387 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:24 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:24.387 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:24 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:24 smithi053 ceph-mon[29182]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:24 smithi053 ceph-mon[29182]: from='osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:44:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:24 smithi053 ceph-mon[29182]: osdmap e25: 4 total, 3 up, 4 in 2024-04-28T12:44:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:24 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:24 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:24.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-04-28T12:44:24.972 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] boot 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T12:44:25.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:25.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:25.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:25.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:25.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:44:25.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:44:25.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:25.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] boot 2024-04-28T12:44:25.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T12:44:25.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:25.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.291 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: osd.3 [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] boot 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:25.292 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:25.293 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:25.293 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:25.293 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:25.293 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:26.178 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:26 smithi077 ceph-mon[32553]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:26.178 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:26 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi053 2024-04-28T12:44:26.178 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:26 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi053 to 8546M 2024-04-28T12:44:26.178 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:26 smithi077 ceph-mon[32553]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T12:44:26.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:26 smithi191 ceph-mon[33016]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:26.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:26 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi053 2024-04-28T12:44:26.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:26 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi053 to 8546M 2024-04-28T12:44:26.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:26 smithi191 ceph-mon[33016]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T12:44:26.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:26 smithi053 ceph-mon[29182]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:44:26.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:26 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi053 2024-04-28T12:44:26.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:26 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi053 to 8546M 2024-04-28T12:44:26.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:26 smithi053 ceph-mon[29182]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T12:44:26.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/1909318346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 msgr2=0x7f2c38103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:26.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/1909318346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c38103da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009990 tx=0x7f2c28009ca0 comp rx=0 tx=0).stop 2024-04-28T12:44:26.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/1909318346 shutdown_connections 2024-04-28T12:44:26.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/1909318346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c38102fa0 0x7f2c3810a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:26.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/1909318346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c38103da0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/1909318346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381025e0 0x7f2c38102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/1909318346 >> 172.21.15.77:0/1909318346 conn(0x7f2c380fd0b0 msgr2=0x7f2c380ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/1909318346 shutdown_connections 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.645+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/1909318346 wait complete. 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 Processor -- start 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 -- start start 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381025e0 0x7f2c38102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c38102fa0 0x7f2c3819e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c381a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810de50 con 0x7f2c381025e0 2024-04-28T12:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810dcd0 con 0x7f2c38103980 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3ea3e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810dfd0 con 0x7f2c38102fa0 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.646+0000 7f2c3cfdb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c381a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c381a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56614/0 (socket says 172.21.15.77:56614) 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 -- 172.21.15.77:0/2319391727 learned_addr learned my addr 172.21.15.77:0/2319391727 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c377fe700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c38102fa0 0x7f2c3819e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c37fff700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381025e0 0x7f2c38102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 -- 172.21.15.77:0/2319391727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c38102fa0 msgr2=0x7f2c3819e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c38102fa0 0x7f2c3819e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 -- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381025e0 msgr2=0x7f2c38102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381025e0 0x7f2c38102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 -- 172.21.15.77:0/2319391727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c381a1240 con 0x7f2c38103980 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c37fff700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381025e0 0x7f2c38102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3cfdb700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c381a0970 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000b970 tx=0x7f2c3000bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c357fa700 1 -- 172.21.15.77:0/2319391727 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30019070 con 0x7f2c38103980 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c357fa700 1 -- 172.21.15.77:0/2319391727 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c30014440 con 0x7f2c38103980 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c357fa700 1 -- 172.21.15.77:0/2319391727 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000ce60 con 0x7f2c38103980 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c381a1530 con 0x7f2c38103980 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.647+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c381a1aa0 con 0x7f2c38103980 2024-04-28T12:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.648+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c3806a110 con 0x7f2c38103980 2024-04-28T12:44:26.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.649+0000 7f2c357fa700 1 -- 172.21.15.77:0/2319391727 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c3001c070 con 0x7f2c38103980 2024-04-28T12:44:26.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.650+0000 7f2c357fa700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c200a3c40 0x7f2c200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:26.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.650+0000 7f2c357fa700 1 -- 172.21.15.77:0/2319391727 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7f2c300ce630 con 0x7f2c38103980 2024-04-28T12:44:26.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.650+0000 7f2c37fff700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c200a3c40 0x7f2c200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:26.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.650+0000 7f2c37fff700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c200a3c40 0x7f2c200a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006010 tx=0x7f2c280095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:26.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.654+0000 7f2c357fa700 1 -- 172.21.15.77:0/2319391727 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2c3009c230 con 0x7f2c38103980 2024-04-28T12:44:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:26.860+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2c38063860 con 0x7f2c200a3c40 2024-04-28T12:44:27.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:27 smithi077 ceph-mon[32553]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:44:27.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:27 smithi077 ceph-mon[32553]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T12:44:27.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:27 smithi077 ceph-mon[32553]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:27.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:27 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:27.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:27 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:27.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:27 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:27 smithi191 ceph-mon[33016]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:44:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:27 smithi191 ceph-mon[33016]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T12:44:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:27 smithi191 ceph-mon[33016]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:27 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:27 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:27 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:27.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:27 smithi053 ceph-mon[29182]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:44:27.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:27 smithi053 ceph-mon[29182]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T12:44:27.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:27 smithi053 ceph-mon[29182]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:27.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:27 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:27.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:27 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:27.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:27 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:28 smithi077 ceph-mon[32553]: osdmap e30: 4 total, 4 up, 4 in 2024-04-28T12:44:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:28 smithi191 ceph-mon[33016]: osdmap e30: 4 total, 4 up, 4 in 2024-04-28T12:44:28.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:28 smithi053 ceph-mon[29182]: osdmap e30: 4 total, 4 up, 4 in 2024-04-28T12:44:29.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:29 smithi077 ceph-mon[32553]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:44:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:29 smithi191 ceph-mon[33016]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:44:29.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:29 smithi053 ceph-mon[29182]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:44:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:30 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/2703058890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]: dispatch 2024-04-28T12:44:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:30 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]: dispatch 2024-04-28T12:44:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:30 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]': finished 2024-04-28T12:44:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:30 smithi077 ceph-mon[32553]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T12:44:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:30 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/1070221891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:30 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/2703058890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]: dispatch 2024-04-28T12:44:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:30 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]: dispatch 2024-04-28T12:44:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:30 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]': finished 2024-04-28T12:44:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:30 smithi191 ceph-mon[33016]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T12:44:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:30.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:30 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/1070221891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:30.796 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:30 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/2703058890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]: dispatch 2024-04-28T12:44:30.796 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:30 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]: dispatch 2024-04-28T12:44:30.796 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:30 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "990ad0bc-63b6-4466-94ca-75ba51801c7c"}]': finished 2024-04-28T12:44:30.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:30 smithi053 ceph-mon[29182]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T12:44:30.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:30.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:30 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/1070221891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:31 smithi053 ceph-mon[29182]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-28T12:44:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:31 smithi077 ceph-mon[32553]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-28T12:44:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:31 smithi191 ceph-mon[33016]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-28T12:44:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:33 smithi053 ceph-mon[29182]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:33.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:33 smithi077 ceph-mon[32553]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:33 smithi191 ceph-mon[33016]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:35.880 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:35 smithi077 ceph-mon[32553]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:35 smithi053 ceph-mon[29182]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:35 smithi191 ceph-mon[33016]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:37 smithi053 ceph-mon[29182]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T12:44:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:37 smithi053 ceph-mon[29182]: Deploying daemon osd.4 on smithi077 2024-04-28T12:44:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:37 smithi077 ceph-mon[32553]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T12:44:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:37 smithi077 ceph-mon[32553]: Deploying daemon osd.4 on smithi077 2024-04-28T12:44:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:37 smithi191 ceph-mon[33016]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T12:44:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:37 smithi191 ceph-mon[33016]: Deploying daemon osd.4 on smithi077 2024-04-28T12:44:39.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:39 smithi077 ceph-mon[32553]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:39 smithi053 ceph-mon[29182]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:39 smithi191 ceph-mon[33016]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:41.787 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:41 smithi077 ceph-mon[32553]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:41.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:41 smithi053 ceph-mon[29182]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:41 smithi191 ceph-mon[33016]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:43 smithi077 ceph-mon[32553]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:43 smithi191 ceph-mon[33016]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:44.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:43 smithi053 ceph-mon[29182]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:44.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:44.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:44.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:45 smithi077 ceph-mon[32553]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:45 smithi191 ceph-mon[33016]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:46.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:45 smithi053 ceph-mon[29182]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:46.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.178+0000 7f2c357fa700 1 -- 172.21.15.77:0/2319391727 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2c38063860 con 0x7f2c200a3c40 2024-04-28T12:44:46.174 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 4 on host 'smithi077' 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c200a3c40 msgr2=0x7f2c200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c200a3c40 0x7f2c200a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006010 tx=0x7f2c280095f0 comp rx=0 tx=0).stop 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 msgr2=0x7f2c381a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c381a0970 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000b970 tx=0x7f2c3000bd30 comp rx=0 tx=0).stop 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 shutdown_connections 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c200a3c40 0x7f2c200a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c381025e0 0x7f2c38102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c38103980 0x7f2c381a0970 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:46.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 --2- 172.21.15.77:0/2319391727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c38102fa0 0x7f2c3819e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:46.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 >> 172.21.15.77:0/2319391727 conn(0x7f2c380fd0b0 msgr2=0x7f2c381069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:46.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 shutdown_connections 2024-04-28T12:44:46.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:44:46.180+0000 7f2c3ea3e700 1 -- 172.21.15.77:0/2319391727 wait complete. 2024-04-28T12:44:46.885 DEBUG:teuthology.orchestra.run.smithi077:osd.4> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.4.service 2024-04-28T12:44:46.887 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/vg_nvme/lv_3... 2024-04-28T12:44:46.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T12:44:47.134 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:44:47.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:47 smithi191 ceph-mon[33016]: from='osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:47.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:47 smithi191 ceph-mon[33016]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:47.389 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.389 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:47 smithi077 ceph-mon[32553]: from='osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:47.390 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:47 smithi077 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:47.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:47.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:47 smithi053 ceph-mon[29182]: from='osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:47.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:47 smithi053 ceph-mon[29182]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:48.361 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-mon[32553]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:48.361 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T12:44:48.361 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-mon[32553]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T12:44:48.361 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:48.361 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-mon[32553]: from='osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:44:48.362 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:44:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:48 smithi053 ceph-mon[29182]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:48 smithi053 ceph-mon[29182]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T12:44:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:48 smithi053 ceph-mon[29182]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T12:44:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:48 smithi053 ceph-mon[29182]: from='osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:44:48.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:48 smithi053 ceph-mon[29182]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:44:48.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:48 smithi191 ceph-mon[33016]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:48.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:48 smithi191 ceph-mon[33016]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T12:44:48.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:48 smithi191 ceph-mon[33016]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T12:44:48.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:48.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:48 smithi191 ceph-mon[33016]: from='osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:44:48.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:48 smithi191 ceph-mon[33016]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:44:49.087 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T12:44:49.087 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T12:44:49.087 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-28T12:44:49.087 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-28T12:44:49.087 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113506 s, 924 MB/s 2024-04-28T12:44:49.087 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 2024-04-28T12:44:49.087 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-28T12:44:49.198 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4[44080]: 2024-04-28T12:44:48.898+0000 7f27ee1c9700 -1 osd.4 0 waiting for initial osdmap 2024-04-28T12:44:49.198 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 12:44:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4[44080]: 2024-04-28T12:44:48.905+0000 7f27e57b3700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:44:49.592 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:49 smithi077 ceph-mon[32553]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:49.593 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:49 smithi077 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:44:49.593 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:49 smithi077 ceph-mon[32553]: osdmap e33: 5 total, 4 up, 5 in 2024-04-28T12:44:49.593 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:49.593 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:49.593 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:49 smithi077 ceph-mon[32553]: from='osd.4 ' entity='osd.4' 2024-04-28T12:44:49.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:49 smithi191 ceph-mon[33016]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:49.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:49 smithi191 ceph-mon[33016]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:44:49.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:49 smithi191 ceph-mon[33016]: osdmap e33: 5 total, 4 up, 5 in 2024-04-28T12:44:49.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:49.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:49.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:49 smithi191 ceph-mon[33016]: from='osd.4 ' entity='osd.4' 2024-04-28T12:44:49.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2024-04-28T12:44:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:49 smithi053 ceph-mon[29182]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:49 smithi053 ceph-mon[29182]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:44:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:49 smithi053 ceph-mon[29182]: osdmap e33: 5 total, 4 up, 5 in 2024-04-28T12:44:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:49 smithi053 ceph-mon[29182]: from='osd.4 ' entity='osd.4' 2024-04-28T12:44:50.052 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:44:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:44:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] boot 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:44:50.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:44:50.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:50.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] boot 2024-04-28T12:44:50.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T12:44:50.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:50.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: osd.4 [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] boot 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:50.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:50.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:50.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:50.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:51.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 -- 172.21.15.191:0/3014194318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194103580 msgr2=0x7f71941039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:51.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 --2- 172.21.15.191:0/3014194318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194103580 0x7f71941039e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7184009990 tx=0x7f7184009ca0 comp rx=0 tx=0).stop 2024-04-28T12:44:51.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 -- 172.21.15.191:0/3014194318 shutdown_connections 2024-04-28T12:44:51.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 --2- 172.21.15.191:0/3014194318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7194102380 0x7f71941027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:51.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 --2- 172.21.15.191:0/3014194318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194103f20 0x7f719410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 --2- 172.21.15.191:0/3014194318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194103580 0x7f71941039e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 -- 172.21.15.191:0/3014194318 >> 172.21.15.191:0/3014194318 conn(0x7f71940fd940 msgr2=0x7f71940ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 -- 172.21.15.191:0/3014194318 shutdown_connections 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 -- 172.21.15.191:0/3014194318 wait complete. 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 Processor -- start 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.683+0000 7f719c089700 1 -- start start 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194102380 0x7f7194199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7194103580 0x7f7194199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194103f20 0x7f71941a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f719410de20 con 0x7f7194103580 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f719410dca0 con 0x7f7194103f20 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f719410dfa0 con 0x7f7194102380 2024-04-28T12:44:51.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194102380 0x7f7194199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194102380 0x7f7194199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:39608/0 (socket says 172.21.15.191:39608) 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 -- 172.21.15.191:0/1023809855 learned_addr learned my addr 172.21.15.191:0/1023809855 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199624700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7194103580 0x7f7194199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719a626700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194103f20 0x7f71941a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 -- 172.21.15.191:0/1023809855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194103f20 msgr2=0x7f71941a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194103f20 0x7f71941a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 -- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7194103580 msgr2=0x7f7194199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7194103580 0x7f7194199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 -- 172.21.15.191:0/1023809855 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71941a1290 con 0x7f7194102380 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719a626700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194103f20 0x7f71941a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f7199e25700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194102380 0x7f7194199630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f719000d9b0 tx=0x7f719000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f718affd700 1 -- 172.21.15.191:0/1023809855 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7190016070 con 0x7f7194102380 2024-04-28T12:44:51.682 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f718affd700 1 -- 172.21.15.191:0/1023809855 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719000be00 con 0x7f7194102380 2024-04-28T12:44:51.683 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71941a1580 con 0x7f7194102380 2024-04-28T12:44:51.683 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f718affd700 1 -- 172.21.15.191:0/1023809855 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719000f3c0 con 0x7f7194102380 2024-04-28T12:44:51.683 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.684+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71941a1ac0 con 0x7f7194102380 2024-04-28T12:44:51.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.685+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719406a110 con 0x7f7194102380 2024-04-28T12:44:51.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.686+0000 7f718affd700 1 -- 172.21.15.191:0/1023809855 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7190019070 con 0x7f7194102380 2024-04-28T12:44:51.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.687+0000 7f718affd700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f71800a3c40 0x7f71800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:51.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.687+0000 7f718affd700 1 -- 172.21.15.191:0/1023809855 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f71900cf780 con 0x7f7194102380 2024-04-28T12:44:51.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.687+0000 7f7199624700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f71800a3c40 0x7f71800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:51.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.688+0000 7f7199624700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f71800a3c40 0x7f71800a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f718400b5c0 tx=0x7f7184005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:51.686 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.691+0000 7f718affd700 1 -- 172.21.15.191:0/1023809855 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7190020020 con 0x7f7194102380 2024-04-28T12:44:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:51 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi077 2024-04-28T12:44:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:51 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi077 to 8544M 2024-04-28T12:44:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:51 smithi191 ceph-mon[33016]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:51.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:44:51.899+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f71940ffb40 con 0x7f71800a3c40 2024-04-28T12:44:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:51 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi077 2024-04-28T12:44:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:51 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi077 to 8544M 2024-04-28T12:44:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:51 smithi053 ceph-mon[29182]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:51 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi077 2024-04-28T12:44:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:51 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi077 to 8544M 2024-04-28T12:44:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:51 smithi077 ceph-mon[32553]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:44:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:52 smithi191 ceph-mon[33016]: osdmap e35: 5 total, 5 up, 5 in 2024-04-28T12:44:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:52 smithi191 ceph-mon[33016]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:52.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:52.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:52 smithi053 ceph-mon[29182]: osdmap e35: 5 total, 5 up, 5 in 2024-04-28T12:44:52.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:52 smithi053 ceph-mon[29182]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:52.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:52.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:52.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:52 smithi077 ceph-mon[32553]: osdmap e35: 5 total, 5 up, 5 in 2024-04-28T12:44:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:52 smithi077 ceph-mon[32553]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:53.739 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:53 smithi191 ceph-mon[33016]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:53.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:53 smithi053 ceph-mon[29182]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:53.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:53 smithi077 ceph-mon[32553]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:55 smithi191 ceph-mon[33016]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:55 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/2246692096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]: dispatch 2024-04-28T12:44:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:55 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]: dispatch 2024-04-28T12:44:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:55 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]': finished 2024-04-28T12:44:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:55 smithi191 ceph-mon[33016]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T12:44:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:55.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:55 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/265699497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:55 smithi053 ceph-mon[29182]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:55 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/2246692096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]: dispatch 2024-04-28T12:44:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:55 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]: dispatch 2024-04-28T12:44:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:55 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]': finished 2024-04-28T12:44:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:55 smithi053 ceph-mon[29182]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T12:44:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:55 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/265699497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:55 smithi077 ceph-mon[32553]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:55 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/2246692096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]: dispatch 2024-04-28T12:44:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:55 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]: dispatch 2024-04-28T12:44:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:55 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3015dc0f-1a0e-4fc9-b38c-f779c500f3b5"}]': finished 2024-04-28T12:44:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:55 smithi077 ceph-mon[32553]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T12:44:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:55 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/265699497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:57 smithi191 ceph-mon[33016]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:57.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:57 smithi053 ceph-mon[29182]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:57.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:57 smithi077 ceph-mon[32553]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:59.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:44:59 smithi191 ceph-mon[33016]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:59.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:44:59 smithi053 ceph-mon[29182]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:44:59 smithi077 ceph-mon[32553]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:01 smithi191 ceph-mon[33016]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:01.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:01 smithi053 ceph-mon[29182]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:01 smithi077 ceph-mon[32553]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T12:45:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:02 smithi191 ceph-mon[33016]: Deploying daemon osd.5 on smithi191 2024-04-28T12:45:02.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T12:45:02.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:02.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:02 smithi053 ceph-mon[29182]: Deploying daemon osd.5 on smithi191 2024-04-28T12:45:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T12:45:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:02.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:02 smithi077 ceph-mon[32553]: Deploying daemon osd.5 on smithi191 2024-04-28T12:45:03.762 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:03 smithi191 ceph-mon[33016]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:03.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:03 smithi053 ceph-mon[29182]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:03.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:03 smithi077 ceph-mon[32553]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:05.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:05 smithi191 ceph-mon[33016]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:05.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:05 smithi053 ceph-mon[29182]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:05.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:05 smithi077 ceph-mon[32553]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:07 smithi191 ceph-mon[33016]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:07.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:07 smithi053 ceph-mon[29182]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:07.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:07 smithi077 ceph-mon[32553]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:08.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:08.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:08.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:08.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:08.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:08.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:08.960 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:08.960 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:08.960 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:09.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:09 smithi053 ceph-mon[29182]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:09.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:09 smithi077 ceph-mon[32553]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:09 smithi191 ceph-mon[33016]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:11.385 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.385 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.385 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:11.385 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:11.385 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.385 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:11 smithi191 ceph-mon[33016]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:11.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.469+0000 7f718affd700 1 -- 172.21.15.191:0/1023809855 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f71940ffb40 con 0x7f71800a3c40 2024-04-28T12:45:11.466 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2024-04-28T12:45:11.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f71800a3c40 msgr2=0x7f71800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:11.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f71800a3c40 0x7f71800a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f718400b5c0 tx=0x7f7184005fb0 comp rx=0 tx=0).stop 2024-04-28T12:45:11.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194102380 msgr2=0x7f7194199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:11.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194102380 0x7f7194199630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f719000d9b0 tx=0x7f719000dcc0 comp rx=0 tx=0).stop 2024-04-28T12:45:11.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 shutdown_connections 2024-04-28T12:45:11.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f71800a3c40 0x7f71800a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:11.469 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194103f20 0x7f71941a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:11.469 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7194102380 0x7f7194199630 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:11.469 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 --2- 172.21.15.191:0/1023809855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7194103580 0x7f7194199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:11.469 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 >> 172.21.15.191:0/1023809855 conn(0x7f71940fd940 msgr2=0x7f71940ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:11.469 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 shutdown_connections 2024-04-28T12:45:11.469 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:45:11.471+0000 7f719c089700 1 -- 172.21.15.191:0/1023809855 wait complete. 2024-04-28T12:45:11.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:11.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:11.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:11 smithi053 ceph-mon[29182]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:11.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:11.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:11.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:11.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:11 smithi077 ceph-mon[32553]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:12.138 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.5.service 2024-04-28T12:45:12.140 INFO:tasks.cephadm:Deploying osd.6 on smithi053 with /dev/vg_nvme/lv_2... 2024-04-28T12:45:12.141 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T12:45:12.419 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:45:12.536 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:12.536 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:12.536 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:12.536 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:12 smithi053 ceph-mon[29182]: from='osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:45:12.536 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:12 smithi053 ceph-mon[29182]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:45:12.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:12.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:12.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:12.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:12 smithi077 ceph-mon[32553]: from='osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:45:12.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:12 smithi077 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:45:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:12 smithi191 ceph-mon[33016]: from='osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:45:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:12 smithi191 ceph-mon[33016]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:45:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-mon[33016]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-mon[33016]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T12:45:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-mon[33016]: from='osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:45:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-mon[33016]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T12:45:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:13.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-mon[33016]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:45:13.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:13 smithi077 ceph-mon[32553]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:13 smithi077 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T12:45:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:13 smithi077 ceph-mon[32553]: from='osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:45:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:13 smithi077 ceph-mon[32553]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T12:45:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:13 smithi077 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:45:13.714 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:13 smithi053 ceph-mon[29182]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:13.715 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:13 smithi053 ceph-mon[29182]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T12:45:13.715 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:13 smithi053 ceph-mon[29182]: from='osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:45:13.715 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:13 smithi053 ceph-mon[29182]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T12:45:13.715 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:13.715 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:13 smithi053 ceph-mon[29182]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:45:14.287 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T12:45:13.896+0000 7f3ba4288700 -1 osd.5 0 waiting for initial osdmap 2024-04-28T12:45:14.287 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 12:45:13 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T12:45:13.904+0000 7f3b9c073700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:45:14.433 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T12:45:14.433 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T12:45:14.433 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-28T12:45:14.433 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-28T12:45:14.433 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00829794 s, 1.3 GB/s 2024-04-28T12:45:14.433 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: osdmap e38: 6 total, 5 up, 6 in 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: from='osd.5 ' entity='osd.5' 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] boot 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T12:45:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:14 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: osdmap e38: 6 total, 5 up, 6 in 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: from='osd.5 ' entity='osd.5' 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] boot 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T12:45:14.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:14 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.742 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:45:14.742 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: osdmap e38: 6 total, 5 up, 6 in 2024-04-28T12:45:14.742 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.742 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:14.742 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: from='osd.5 ' entity='osd.5' 2024-04-28T12:45:14.742 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: osd.5 [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] boot 2024-04-28T12:45:14.743 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T12:45:14.743 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:14 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:45:15.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_2 2024-04-28T12:45:15.370 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:45:15.696 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:15 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:45:15.696 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:15 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:45:15.696 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:15 smithi053 ceph-mon[29182]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:15.696 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:15 smithi053 ceph-mon[29182]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T12:45:15.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:15 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:45:15.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:15 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:45:15.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:15 smithi077 ceph-mon[32553]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:15.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:15 smithi077 ceph-mon[32553]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T12:45:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:15 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:45:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:15 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:45:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:15 smithi191 ceph-mon[33016]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:45:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:15 smithi191 ceph-mon[33016]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T12:45:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi191 2024-04-28T12:45:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi191 to 8544M 2024-04-28T12:45:16.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:16.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:16.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:16 smithi077 ceph-mon[32553]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T12:45:16.759 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi191 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi191 to 8544M 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.760 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:16 smithi053 ceph-mon[29182]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi191 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi191 to 8544M 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:16 smithi191 ceph-mon[33016]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T12:45:17.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.112+0000 7f83d111b700 1 -- 172.21.15.53:0/359969229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1034a0 msgr2=0x7f83cc1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:17.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.112+0000 7f83d111b700 1 --2- 172.21.15.53:0/359969229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1034a0 0x7f83cc1038c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0099c0 tx=0x7f83bc009cd0 comp rx=0 tx=0).stop 2024-04-28T12:45:17.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.112+0000 7f83d111b700 1 -- 172.21.15.53:0/359969229 shutdown_connections 2024-04-28T12:45:17.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.112+0000 7f83d111b700 1 --2- 172.21.15.53:0/359969229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1034a0 0x7f83cc1038c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:17.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.112+0000 7f83d111b700 1 --2- 172.21.15.53:0/359969229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83cc105040 0x7f83cc10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:17.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.112+0000 7f83d111b700 1 --2- 172.21.15.53:0/359969229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83cc1046a0 0x7f83cc104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:17.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.112+0000 7f83d111b700 1 -- 172.21.15.53:0/359969229 >> 172.21.15.53:0/359969229 conn(0x7f83cc0fea40 msgr2=0x7f83cc100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:17.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.113+0000 7f83d111b700 1 -- 172.21.15.53:0/359969229 shutdown_connections 2024-04-28T12:45:17.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.113+0000 7f83d111b700 1 -- 172.21.15.53:0/359969229 wait complete. 2024-04-28T12:45:17.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.113+0000 7f83d111b700 1 Processor -- start 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.113+0000 7f83d111b700 1 -- start start 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83d111b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83cc1034a0 0x7f83cc1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83d111b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1046a0 0x7f83cc199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83d111b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83cc105040 0x7f83cc1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83d111b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc10ef30 con 0x7f83cc1046a0 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83d111b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc10edb0 con 0x7f83cc1034a0 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83d111b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc10f0b0 con 0x7f83cc105040 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1046a0 0x7f83cc199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:17.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1046a0 0x7f83cc199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32792/0 (socket says 172.21.15.53:32792) 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 -- 172.21.15.53:0/1917302123 learned_addr learned my addr 172.21.15.53:0/1917302123 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83cb59e700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83cc105040 0x7f83cc1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83cad9d700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83cc1034a0 0x7f83cc1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 -- 172.21.15.53:0/1917302123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83cc105040 msgr2=0x7f83cc1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83cc105040 0x7f83cc1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 -- 172.21.15.53:0/1917302123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83cc1034a0 msgr2=0x7f83cc1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83cc1034a0 0x7f83cc1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 -- 172.21.15.53:0/1917302123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83cc075650 con 0x7f83cc1046a0 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83cad9d700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83cc1034a0 0x7f83cc1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.114+0000 7f83ca59c700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1046a0 0x7f83cc199b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f83c000e9d0 tx=0x7f83c000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.115+0000 7f83b3fff700 1 -- 172.21.15.53:0/1917302123 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c000cbd0 con 0x7f83cc1046a0 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.115+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83cc075940 con 0x7f83cc1046a0 2024-04-28T12:45:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.115+0000 7f83b3fff700 1 -- 172.21.15.53:0/1917302123 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83c000cd70 con 0x7f83cc1046a0 2024-04-28T12:45:17.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.115+0000 7f83b3fff700 1 -- 172.21.15.53:0/1917302123 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0018680 con 0x7f83cc1046a0 2024-04-28T12:45:17.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.115+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83cc075e80 con 0x7f83cc1046a0 2024-04-28T12:45:17.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.116+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83cc06a110 con 0x7f83cc1046a0 2024-04-28T12:45:17.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.117+0000 7f83b3fff700 1 -- 172.21.15.53:0/1917302123 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f83c0010430 con 0x7f83cc1046a0 2024-04-28T12:45:17.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.118+0000 7f83b3fff700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f83b40a3c40 0x7f83b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:17.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.118+0000 7f83cad9d700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f83b40a3c40 0x7f83b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:17.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.118+0000 7f83b3fff700 1 -- 172.21.15.53:0/1917302123 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f83c00cfcb0 con 0x7f83cc1046a0 2024-04-28T12:45:17.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.118+0000 7f83cad9d700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f83b40a3c40 0x7f83b40a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0036f0 tx=0x7f83bc0037c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:17.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.121+0000 7f83b3fff700 1 -- 172.21.15.53:0/1917302123 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f83c009d490 con 0x7f83cc1046a0 2024-04-28T12:45:17.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:17.338+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f83cc10a4d0 con 0x7f83b40a3c40 2024-04-28T12:45:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:17 smithi191 ceph-mon[33016]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 199 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:17 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:17 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:17 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:17.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:17 smithi053 ceph-mon[29182]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 199 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:17.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:17 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:17.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:17 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:17.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:17 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:17.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:17 smithi077 ceph-mon[32553]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 199 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:17.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:17 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:17.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:17 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:17.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:17 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:18.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:18 smithi053 ceph-mon[29182]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:18.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:18 smithi053 ceph-mon[29182]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T12:45:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:18 smithi191 ceph-mon[33016]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:18 smithi191 ceph-mon[33016]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T12:45:18.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:18 smithi077 ceph-mon[32553]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:18.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:18 smithi077 ceph-mon[32553]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T12:45:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:19 smithi053 ceph-mon[29182]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 199 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:19 smithi053 ceph-mon[29182]: osdmap e43: 6 total, 6 up, 6 in 2024-04-28T12:45:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:19 smithi191 ceph-mon[33016]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 199 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:19 smithi191 ceph-mon[33016]: osdmap e43: 6 total, 6 up, 6 in 2024-04-28T12:45:19.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:19 smithi077 ceph-mon[32553]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 199 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:19.861 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:19 smithi077 ceph-mon[32553]: osdmap e43: 6 total, 6 up, 6 in 2024-04-28T12:45:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:20 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1717009787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2591cbe0-d151-45b4-87c1-532c13a53037"}]: dispatch 2024-04-28T12:45:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:20 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1717009787' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2591cbe0-d151-45b4-87c1-532c13a53037"}]': finished 2024-04-28T12:45:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:20 smithi191 ceph-mon[33016]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T12:45:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:20 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:20.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:20 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1717009787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2591cbe0-d151-45b4-87c1-532c13a53037"}]: dispatch 2024-04-28T12:45:20.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:20 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1717009787' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2591cbe0-d151-45b4-87c1-532c13a53037"}]': finished 2024-04-28T12:45:20.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:20 smithi053 ceph-mon[29182]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T12:45:20.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:20 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:20.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:20 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1717009787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2591cbe0-d151-45b4-87c1-532c13a53037"}]: dispatch 2024-04-28T12:45:20.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:20 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1717009787' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2591cbe0-d151-45b4-87c1-532c13a53037"}]': finished 2024-04-28T12:45:20.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:20 smithi077 ceph-mon[32553]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T12:45:20.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:20 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:21 smithi191 ceph-mon[33016]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:21.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:21 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3008369827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:21.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:21 smithi053 ceph-mon[29182]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:21.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:21 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3008369827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:21.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:21 smithi077 ceph-mon[32553]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:21 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3008369827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:23 smithi191 ceph-mon[33016]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:45:23.792 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:23 smithi053 ceph-mon[29182]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:45:23.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:23 smithi077 ceph-mon[32553]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:45:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:25 smithi191 ceph-mon[33016]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-28T12:45:25.820 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:25 smithi053 ceph-mon[29182]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-28T12:45:25.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:25 smithi077 ceph-mon[32553]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-28T12:45:27.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:27 smithi053 ceph-mon[29182]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:45:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:27 smithi191 ceph-mon[33016]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:45:27.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:27 smithi077 ceph-mon[32553]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:45:28.629 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T12:45:28.629 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:28.629 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:28 smithi053 ceph-mon[29182]: Deploying daemon osd.6 on smithi053 2024-04-28T12:45:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T12:45:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:28 smithi191 ceph-mon[33016]: Deploying daemon osd.6 on smithi053 2024-04-28T12:45:28.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T12:45:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:28 smithi077 ceph-mon[32553]: Deploying daemon osd.6 on smithi053 2024-04-28T12:45:29.686 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:29 smithi053 ceph-mon[29182]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T12:45:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:29 smithi191 ceph-mon[33016]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T12:45:29.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:29 smithi077 ceph-mon[32553]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T12:45:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:31 smithi191 ceph-mon[33016]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-28T12:45:31.813 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:31 smithi053 ceph-mon[29182]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-28T12:45:31.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:31 smithi077 ceph-mon[32553]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-28T12:45:33.614 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:33 smithi053 ceph-mon[29182]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-28T12:45:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:33 smithi191 ceph-mon[33016]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-28T12:45:33.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:33 smithi077 ceph-mon[32553]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-28T12:45:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:35.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:35.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:35.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:35.756 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:35 smithi053 ceph-mon[29182]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:35.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:35 smithi077 ceph-mon[32553]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:35 smithi191 ceph-mon[33016]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.449+0000 7f83b3fff700 1 -- 172.21.15.53:0/1917302123 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f83cc10a4d0 con 0x7f83b40a3c40 2024-04-28T12:45:37.446 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 6 on host 'smithi053' 2024-04-28T12:45:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f83b40a3c40 msgr2=0x7f83b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f83b40a3c40 0x7f83b40a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0036f0 tx=0x7f83bc0037c0 comp rx=0 tx=0).stop 2024-04-28T12:45:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1046a0 msgr2=0x7f83cc199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1046a0 0x7f83cc199b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f83c000e9d0 tx=0x7f83c000ece0 comp rx=0 tx=0).stop 2024-04-28T12:45:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 shutdown_connections 2024-04-28T12:45:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f83b40a3c40 0x7f83b40a6100 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83cc105040 0x7f83cc1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:37.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83cc1034a0 0x7f83cc1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:37.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 --2- 172.21.15.53:0/1917302123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83cc1046a0 0x7f83cc199b20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:37.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 >> 172.21.15.53:0/1917302123 conn(0x7f83cc0fea40 msgr2=0x7f83cc107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:37.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 shutdown_connections 2024-04-28T12:45:37.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:45:37.452+0000 7f83d111b700 1 -- 172.21.15.53:0/1917302123 wait complete. 2024-04-28T12:45:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:37 smithi077 ceph-mon[32553]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:37 smithi077 ceph-mon[32553]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T12:45:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:37 smithi191 ceph-mon[33016]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:37 smithi191 ceph-mon[33016]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T12:45:38.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:37 smithi053 ceph-mon[29182]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:38.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:38.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:38.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:37 smithi053 ceph-mon[29182]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T12:45:38.176 DEBUG:teuthology.orchestra.run.smithi053:osd.6> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.6.service 2024-04-28T12:45:38.178 INFO:tasks.cephadm:Deploying osd.7 on smithi077 with /dev/vg_nvme/lv_2... 2024-04-28T12:45:38.179 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T12:45:38.424 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:45:39.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:39.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:39.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-mon[29182]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T12:45:39.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-mon[29182]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T12:45:39.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-mon[29182]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:45:39.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:39 smithi191 ceph-mon[33016]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T12:45:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:39 smithi191 ceph-mon[33016]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T12:45:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:39 smithi191 ceph-mon[33016]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:45:39.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:39.580 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:39.580 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:39.580 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:39 smithi077 ceph-mon[32553]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T12:45:39.580 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:39 smithi077 ceph-mon[32553]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T12:45:39.580 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:39 smithi077 ceph-mon[32553]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:45:39.580 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:40.237 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T12:45:39.924+0000 7f6fceb69700 -1 osd.6 0 waiting for initial osdmap 2024-04-28T12:45:40.237 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 12:45:39 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T12:45:39.935+0000 7f6fc6954700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:45:40.362 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:40 smithi077 ceph-mon[32553]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:40.362 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:40 smithi077 ceph-mon[32553]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:45:40.362 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:40 smithi077 ceph-mon[32553]: osdmap e46: 7 total, 6 up, 7 in 2024-04-28T12:45:40.362 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:40 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:40.362 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:40 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:40.362 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:40 smithi077 ceph-mon[32553]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' 2024-04-28T12:45:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:40 smithi191 ceph-mon[33016]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:40 smithi191 ceph-mon[33016]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:45:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:40 smithi191 ceph-mon[33016]: osdmap e46: 7 total, 6 up, 7 in 2024-04-28T12:45:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:40 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:40 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:40.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:40 smithi191 ceph-mon[33016]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' 2024-04-28T12:45:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T12:45:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T12:45:40.564 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-28T12:45:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-28T12:45:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112833 s, 929 MB/s 2024-04-28T12:45:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-28T12:45:40.600 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:40 smithi053 ceph-mon[29182]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:40.600 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:40 smithi053 ceph-mon[29182]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:45:40.600 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:40 smithi053 ceph-mon[29182]: osdmap e46: 7 total, 6 up, 7 in 2024-04-28T12:45:40.600 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:40 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:40.600 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:40 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:40.600 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:40 smithi053 ceph-mon[29182]: from='osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966]' entity='osd.6' 2024-04-28T12:45:41.210 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-04-28T12:45:41.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:41 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:45:41.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:41 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:45:41.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:41.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:41 smithi077 ceph-mon[32553]: osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] boot 2024-04-28T12:45:41.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:41 smithi077 ceph-mon[32553]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T12:45:41.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:41.456 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:45:41.502 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:41 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:45:41.502 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:41 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:45:41.502 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:41.502 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:41 smithi053 ceph-mon[29182]: osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] boot 2024-04-28T12:45:41.502 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:41 smithi053 ceph-mon[29182]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T12:45:41.502 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:41 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:45:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:41 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:45:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:41 smithi191 ceph-mon[33016]: osd.6 [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] boot 2024-04-28T12:45:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:41 smithi191 ceph-mon[33016]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T12:45:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:42.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:42 smithi191 ceph-mon[33016]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:42.623 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:42 smithi077 ceph-mon[32553]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:42.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:42 smithi053 ceph-mon[29182]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:43.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.150+0000 7f1e61d9e700 1 -- 172.21.15.77:0/2100703668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c1045a0 msgr2=0x7f1e5c104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.150+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/2100703668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c1045a0 0x7f1e5c104a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1e480099c0 tx=0x7f1e48009cd0 comp rx=0 tx=0).stop 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.150+0000 7f1e61d9e700 1 -- 172.21.15.77:0/2100703668 shutdown_connections 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.150+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/2100703668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e5c104f40 0x7f1e5c10b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.150+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/2100703668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e5c1033a0 0x7f1e5c1037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.150+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/2100703668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c1045a0 0x7f1e5c104a00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.151+0000 7f1e61d9e700 1 -- 172.21.15.77:0/2100703668 >> 172.21.15.77:0/2100703668 conn(0x7f1e5c0fe960 msgr2=0x7f1e5c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.151+0000 7f1e61d9e700 1 -- 172.21.15.77:0/2100703668 shutdown_connections 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.151+0000 7f1e61d9e700 1 -- 172.21.15.77:0/2100703668 wait complete. 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.151+0000 7f1e61d9e700 1 Processor -- start 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.151+0000 7f1e61d9e700 1 -- start start 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e61d9e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e5c1033a0 0x7f1e5c199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:43.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e61d9e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e5c1045a0 0x7f1e5c199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e61d9e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c104f40 0x7f1e5c1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e61d9e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c10eeb0 con 0x7f1e5c1045a0 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e61d9e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c10ed30 con 0x7f1e5c104f40 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e61d9e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c10f030 con 0x7f1e5c1033a0 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c104f40 0x7f1e5c1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c104f40 0x7f1e5c1a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42404/0 (socket says 172.21.15.77:42404) 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 -- 172.21.15.77:0/680652677 learned_addr learned my addr 172.21.15.77:0/680652677 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e5bfff700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e5c1045a0 0x7f1e5c199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 -- 172.21.15.77:0/680652677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e5c1033a0 msgr2=0x7f1e5c199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e5c1033a0 0x7f1e5c199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 -- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e5c1045a0 msgr2=0x7f1e5c199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e5c1045a0 0x7f1e5c199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 -- 172.21.15.77:0/680652677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c1a1180 con 0x7f1e5c104f40 2024-04-28T12:45:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e5bfff700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e5c1045a0 0x7f1e5c199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:45:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e6159d700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c104f40 0x7f1e5c1a0ae0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e54007ee0 tx=0x7f1e5400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e59ffb700 1 -- 172.21.15.77:0/680652677 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5400dbd0 con 0x7f1e5c104f40 2024-04-28T12:45:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c1a1470 con 0x7f1e5c104f40 2024-04-28T12:45:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e59ffb700 1 -- 172.21.15.77:0/680652677 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e5400dd70 con 0x7f1e5c104f40 2024-04-28T12:45:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.152+0000 7f1e59ffb700 1 -- 172.21.15.77:0/680652677 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54005990 con 0x7f1e5c104f40 2024-04-28T12:45:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.153+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c1a19b0 con 0x7f1e5c104f40 2024-04-28T12:45:43.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.153+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c06a110 con 0x7f1e5c104f40 2024-04-28T12:45:43.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.154+0000 7f1e59ffb700 1 -- 172.21.15.77:0/680652677 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1e5401a020 con 0x7f1e5c104f40 2024-04-28T12:45:43.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.155+0000 7f1e59ffb700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1e4c0a3cc0 0x7f1e4c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:43.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.155+0000 7f1e59ffb700 1 -- 172.21.15.77:0/680652677 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f1e540ceef0 con 0x7f1e5c104f40 2024-04-28T12:45:43.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.155+0000 7f1e60d9c700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1e4c0a3cc0 0x7f1e4c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:43.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.156+0000 7f1e60d9c700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1e4c0a3cc0 0x7f1e4c0a6180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c104400 tx=0x7f1e50006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:43.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.159+0000 7f1e59ffb700 1 -- 172.21.15.77:0/680652677 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1e54010070 con 0x7f1e5c104f40 2024-04-28T12:45:43.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:45:43.367+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1e5c063860 con 0x7f1e4c0a3cc0 2024-04-28T12:45:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: osdmap e48: 7 total, 7 up, 7 in 2024-04-28T12:45:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:43.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: osdmap e48: 7 total, 7 up, 7 in 2024-04-28T12:45:43.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:43.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: osdmap e48: 7 total, 7 up, 7 in 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:43.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:45:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:44 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi053 2024-04-28T12:45:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:44 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi053 to 5697M 2024-04-28T12:45:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:44 smithi053 ceph-mon[29182]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:44 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:44 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:44.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:44 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:44 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi053 2024-04-28T12:45:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:44 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi053 to 5697M 2024-04-28T12:45:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:44 smithi191 ceph-mon[33016]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:44 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:44 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:44.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:44 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:44.658 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:44 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi053 2024-04-28T12:45:44.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:44 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi053 to 5697M 2024-04-28T12:45:44.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:44 smithi077 ceph-mon[32553]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:44.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:44 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:44.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:44 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:44.659 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:44 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:45 smithi191 ceph-mon[33016]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:45.545 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:45 smithi077 ceph-mon[32553]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:45.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:45 smithi053 ceph-mon[29182]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:46.531 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:46 smithi077 ceph-mon[32553]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:46.531 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:46 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/1786351396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]: dispatch 2024-04-28T12:45:46.531 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:46 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]: dispatch 2024-04-28T12:45:46.531 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:46 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]': finished 2024-04-28T12:45:46.531 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:46 smithi077 ceph-mon[32553]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T12:45:46.531 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:46 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:46 smithi191 ceph-mon[33016]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:46 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/1786351396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]: dispatch 2024-04-28T12:45:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:46 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]: dispatch 2024-04-28T12:45:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:46 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]': finished 2024-04-28T12:45:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:46 smithi191 ceph-mon[33016]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T12:45:46.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:46 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:46.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:46 smithi053 ceph-mon[29182]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:46.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:46 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/1786351396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]: dispatch 2024-04-28T12:45:46.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:46 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]: dispatch 2024-04-28T12:45:46.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:46 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "573bb4c3-4e03-48c7-ad54-90cc71ab0311"}]': finished 2024-04-28T12:45:46.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:46 smithi053 ceph-mon[29182]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T12:45:46.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:46 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:47.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:47 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/2051050545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:47.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/2051050545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:47.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:47 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/2051050545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:48.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:48 smithi191 ceph-mon[33016]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:48.598 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:48 smithi077 ceph-mon[32553]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:48.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:48 smithi053 ceph-mon[29182]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:50.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:50 smithi191 ceph-mon[33016]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:50.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:50 smithi053 ceph-mon[29182]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:50.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:50 smithi077 ceph-mon[32553]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:52.377 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:52 smithi077 ceph-mon[32553]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:52.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:52 smithi191 ceph-mon[33016]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:52.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:52 smithi053 ceph-mon[29182]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:54.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:54 smithi077 ceph-mon[32553]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:54.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:54 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T12:45:54.535 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:54 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:54 smithi191 ceph-mon[33016]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:54 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T12:45:54.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:54 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:54.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:54 smithi053 ceph-mon[29182]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:54.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:54 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T12:45:54.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:54 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:55.491 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:55 smithi077 ceph-mon[32553]: Deploying daemon osd.7 on smithi077 2024-04-28T12:45:55.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:55 smithi191 ceph-mon[33016]: Deploying daemon osd.7 on smithi077 2024-04-28T12:45:55.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:55 smithi053 ceph-mon[29182]: Deploying daemon osd.7 on smithi077 2024-04-28T12:45:56.486 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:56 smithi077 ceph-mon[32553]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:56.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:56 smithi191 ceph-mon[33016]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:56.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:56 smithi053 ceph-mon[29182]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:57.467 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:45:57 smithi077 ceph-mon[32553]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:57.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:45:57 smithi053 ceph-mon[29182]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:57.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:45:57 smithi191 ceph-mon[33016]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:00 smithi191 ceph-mon[33016]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:00.571 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:00 smithi077 ceph-mon[32553]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:00.572 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:00.572 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:00.572 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:00 smithi053 ceph-mon[29182]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:02.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:02 smithi077 ceph-mon[32553]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:02.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:02 smithi053 ceph-mon[29182]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:02 smithi191 ceph-mon[33016]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.888+0000 7f1e59ffb700 1 -- 172.21.15.77:0/680652677 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1e5c063860 con 0x7f1e4c0a3cc0 2024-04-28T12:46:02.884 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 7 on host 'smithi077' 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1e4c0a3cc0 msgr2=0x7f1e4c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1e4c0a3cc0 0x7f1e4c0a6180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c104400 tx=0x7f1e50006d20 comp rx=0 tx=0).stop 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c104f40 msgr2=0x7f1e5c1a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c104f40 0x7f1e5c1a0ae0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e54007ee0 tx=0x7f1e5400d3b0 comp rx=0 tx=0).stop 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 shutdown_connections 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1e4c0a3cc0 0x7f1e4c0a6180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e5c104f40 0x7f1e5c1a0ae0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e5c1033a0 0x7f1e5c199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:02.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 --2- 172.21.15.77:0/680652677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e5c1045a0 0x7f1e5c199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:02.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 >> 172.21.15.77:0/680652677 conn(0x7f1e5c0fe960 msgr2=0x7f1e5c107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:02.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 shutdown_connections 2024-04-28T12:46:02.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:46:02.890+0000 7f1e61d9e700 1 -- 172.21.15.77:0/680652677 wait complete. 2024-04-28T12:46:03.539 DEBUG:teuthology.orchestra.run.smithi077:osd.7> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.7.service 2024-04-28T12:46:03.542 INFO:tasks.cephadm:Deploying osd.8 on smithi191 with /dev/vg_nvme/lv_2... 2024-04-28T12:46:03.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T12:46:03.789 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:46:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:03 smithi053 ceph-mon[29182]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:03 smithi053 ceph-mon[29182]: from='osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:46:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:03 smithi053 ceph-mon[29182]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:46:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.156 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:03 smithi191 ceph-mon[33016]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:04.157 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.157 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.157 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:03 smithi191 ceph-mon[33016]: from='osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:46:04.157 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:03 smithi191 ceph-mon[33016]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:46:04.157 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.157 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:03 smithi077 ceph-mon[32553]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:04.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:03 smithi077 ceph-mon[32553]: from='osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:46:04.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:03 smithi077 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:46:04.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:04.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:04 smithi053 ceph-mon[29182]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T12:46:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:04 smithi053 ceph-mon[29182]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T12:46:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:04 smithi053 ceph-mon[29182]: from='osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:46:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:04 smithi053 ceph-mon[29182]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:46:05.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:04 smithi077 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T12:46:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:04 smithi077 ceph-mon[32553]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T12:46:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:04 smithi077 ceph-mon[32553]: from='osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:46:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:04 smithi077 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:46:05.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:04 smithi191 ceph-mon[33016]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T12:46:05.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:04 smithi191 ceph-mon[33016]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T12:46:05.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:05.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:04 smithi191 ceph-mon[33016]: from='osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:46:05.208 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:04 smithi191 ceph-mon[33016]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:46:05.511 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T12:46:05.264+0000 7f1fae1f5700 -1 osd.7 0 waiting for initial osdmap 2024-04-28T12:46:05.512 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T12:46:05.277+0000 7f1fa5fe0700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:46:05.935 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T12:46:05.935 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T12:46:05.935 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-28T12:46:05.935 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-28T12:46:05.935 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113694 s, 922 MB/s 2024-04-28T12:46:05.935 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-28T12:46:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:05 smithi053 ceph-mon[29182]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:05 smithi053 ceph-mon[29182]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:46:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:05 smithi053 ceph-mon[29182]: osdmap e51: 8 total, 7 up, 8 in 2024-04-28T12:46:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:05 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:05 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:05 smithi053 ceph-mon[29182]: from='osd.7 ' entity='osd.7' 2024-04-28T12:46:06.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-mon[32553]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:06.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:46:06.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-mon[32553]: osdmap e51: 8 total, 7 up, 8 in 2024-04-28T12:46:06.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:06.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:06.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:05 smithi077 ceph-mon[32553]: from='osd.7 ' entity='osd.7' 2024-04-28T12:46:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:05 smithi191 ceph-mon[33016]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:05 smithi191 ceph-mon[33016]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:46:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:05 smithi191 ceph-mon[33016]: osdmap e51: 8 total, 7 up, 8 in 2024-04-28T12:46:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:05 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:05 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:06.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:05 smithi191 ceph-mon[33016]: from='osd.7 ' entity='osd.7' 2024-04-28T12:46:06.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2024-04-28T12:46:06.908 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:46:06.945 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:06 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:46:06.945 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:06 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:46:06.945 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:06 smithi191 ceph-mon[33016]: osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] boot 2024-04-28T12:46:06.945 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:06 smithi191 ceph-mon[33016]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T12:46:06.945 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:06.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:06 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:46:06.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:06 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:46:06.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:06 smithi077 ceph-mon[32553]: osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] boot 2024-04-28T12:46:06.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:06 smithi077 ceph-mon[32553]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T12:46:06.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:07.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:06 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:46:07.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:06 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:46:07.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:06 smithi053 ceph-mon[29182]: osd.7 [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] boot 2024-04-28T12:46:07.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:06 smithi053 ceph-mon[29182]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T12:46:07.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi077 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi077 to 5696M 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:07 smithi191 ceph-mon[33016]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T12:46:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T12:46:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi077 2024-04-28T12:46:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi077 to 5696M 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:07 smithi053 ceph-mon[29182]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T12:46:08.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi077 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi077 to 5696M 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:08.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:08.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:08.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:08.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:07 smithi077 ceph-mon[32553]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 -- 172.21.15.191:0/3798295748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158103480 msgr2=0x7f81581038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 --2- 172.21.15.191:0/3798295748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158103480 0x7f81581038a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8148009990 tx=0x7f8148009ca0 comp rx=0 tx=0).stop 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 -- 172.21.15.191:0/3798295748 shutdown_connections 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 --2- 172.21.15.191:0/3798295748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8158105020 0x7f815810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 --2- 172.21.15.191:0/3798295748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158103480 0x7f81581038a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 --2- 172.21.15.191:0/3798295748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8158104680 0x7f8158104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 -- 172.21.15.191:0/3798295748 >> 172.21.15.191:0/3798295748 conn(0x7f81580fea20 msgr2=0x7f8158100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 -- 172.21.15.191:0/3798295748 shutdown_connections 2024-04-28T12:46:08.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 -- 172.21.15.191:0/3798295748 wait complete. 2024-04-28T12:46:08.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.531+0000 7f815d615700 1 Processor -- start 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f815d615700 1 -- start start 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f815d615700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8158103480 0x7f8158199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f815d615700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158104680 0x7f8158199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f815d615700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8158105020 0x7f81581a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f815d615700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f815810ef20 con 0x7f8158103480 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f815d615700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f815810eda0 con 0x7f8158105020 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f815d615700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f815810f0a0 con 0x7f8158104680 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158104680 0x7f8158199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158104680 0x7f8158199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:39414/0 (socket says 172.21.15.191:39414) 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 -- 172.21.15.191:0/3038092703 learned_addr learned my addr 172.21.15.191:0/3038092703 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f8156ffd700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8158103480 0x7f8158199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81577fe700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8158105020 0x7f81581a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 -- 172.21.15.191:0/3038092703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8158105020 msgr2=0x7f81581a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8158105020 0x7f81581a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 -- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8158103480 msgr2=0x7f8158199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8158103480 0x7f8158199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.532+0000 7f81567fc700 1 -- 172.21.15.191:0/3038092703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81581a1260 con 0x7f8158104680 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.533+0000 7f81567fc700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158104680 0x7f8158199b90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f814c00e9d0 tx=0x7f814c00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.533+0000 7f813ffff700 1 -- 172.21.15.191:0/3038092703 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814c00cbd0 con 0x7f8158104680 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.533+0000 7f813ffff700 1 -- 172.21.15.191:0/3038092703 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f814c00ee70 con 0x7f8158104680 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.533+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81581a1550 con 0x7f8158104680 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.533+0000 7f813ffff700 1 -- 172.21.15.191:0/3038092703 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814c018680 con 0x7f8158104680 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.533+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81581a1a90 con 0x7f8158104680 2024-04-28T12:46:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.534+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f815806a110 con 0x7f8158104680 2024-04-28T12:46:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.535+0000 7f813ffff700 1 -- 172.21.15.191:0/3038092703 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f814c010430 con 0x7f8158104680 2024-04-28T12:46:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.535+0000 7f813ffff700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f81400a3c40 0x7f81400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.536+0000 7f813ffff700 1 -- 172.21.15.191:0/3038092703 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f814c0cfdf0 con 0x7f8158104680 2024-04-28T12:46:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.536+0000 7f8156ffd700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f81400a3c40 0x7f81400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.536+0000 7f8156ffd700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f81400a3c40 0x7f81400a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f814800b720 tx=0x7f8148005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.539+0000 7f813ffff700 1 -- 172.21.15.191:0/3038092703 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f814c09d310 con 0x7f8158104680 2024-04-28T12:46:08.740 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:08.744+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8158063860 con 0x7f81400a3c40 2024-04-28T12:46:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:09.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:09.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:09.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:09.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:09.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:09.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:09.871 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:09 smithi191 ceph-mon[33016]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:09.871 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:09 smithi191 ceph-mon[33016]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:09.871 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:09 smithi191 ceph-mon[33016]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T12:46:10.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:09 smithi053 ceph-mon[29182]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:10.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:09 smithi053 ceph-mon[29182]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:10.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:09 smithi053 ceph-mon[29182]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T12:46:10.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:09 smithi077 ceph-mon[32553]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:10.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:09 smithi077 ceph-mon[32553]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:10.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:09 smithi077 ceph-mon[32553]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T12:46:11.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:10 smithi053 ceph-mon[29182]: osdmap e56: 8 total, 8 up, 8 in 2024-04-28T12:46:11.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:10 smithi077 ceph-mon[32553]: osdmap e56: 8 total, 8 up, 8 in 2024-04-28T12:46:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:10 smithi191 ceph-mon[33016]: osdmap e56: 8 total, 8 up, 8 in 2024-04-28T12:46:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:11 smithi053 ceph-mon[29182]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:11 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/3503598804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]: dispatch 2024-04-28T12:46:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:11 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]: dispatch 2024-04-28T12:46:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:11 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]': finished 2024-04-28T12:46:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:11 smithi053 ceph-mon[29182]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T12:46:12.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:12.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:11 smithi077 ceph-mon[32553]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:12.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:11 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/3503598804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]: dispatch 2024-04-28T12:46:12.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:11 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]: dispatch 2024-04-28T12:46:12.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:11 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]': finished 2024-04-28T12:46:12.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:11 smithi077 ceph-mon[32553]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T12:46:12.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:11 smithi191 ceph-mon[33016]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:11 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/3503598804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]: dispatch 2024-04-28T12:46:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:11 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]: dispatch 2024-04-28T12:46:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:11 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44f526cb-dce8-4894-9cd7-21118f8d3052"}]': finished 2024-04-28T12:46:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:11 smithi191 ceph-mon[33016]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T12:46:12.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:13.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:12 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/4069736910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:13.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:12 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/4069736910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:13.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:12 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/4069736910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:14.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:13 smithi053 ceph-mon[29182]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:14.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:13 smithi077 ceph-mon[32553]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:14.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:13 smithi191 ceph-mon[33016]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:16.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:15 smithi053 ceph-mon[29182]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:16.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:15 smithi077 ceph-mon[32553]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:16.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:15 smithi191 ceph-mon[33016]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:17 smithi191 ceph-mon[33016]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:18.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:17 smithi077 ceph-mon[32553]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:18.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:17 smithi053 ceph-mon[29182]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:19.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T12:46:19.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T12:46:19.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:19.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T12:46:19.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:20.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:19 smithi077 ceph-mon[32553]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:20.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:19 smithi077 ceph-mon[32553]: Deploying daemon osd.8 on smithi191 2024-04-28T12:46:20.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:19 smithi191 ceph-mon[33016]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:20.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:19 smithi191 ceph-mon[33016]: Deploying daemon osd.8 on smithi191 2024-04-28T12:46:20.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:19 smithi053 ceph-mon[29182]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:20.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:19 smithi053 ceph-mon[29182]: Deploying daemon osd.8 on smithi191 2024-04-28T12:46:22.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:21 smithi077 ceph-mon[32553]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:22.215 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:21 smithi191 ceph-mon[33016]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:21 smithi053 ceph-mon[29182]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:24.167 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:23 smithi191 ceph-mon[33016]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:24.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:23 smithi077 ceph-mon[32553]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:23 smithi053 ceph-mon[29182]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:25 smithi191 ceph-mon[33016]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:25.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:25.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:25 smithi053 ceph-mon[29182]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:25.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:25.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:25.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:25.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:25 smithi077 ceph-mon[32553]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:25.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:25.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:25.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:27.702 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:27 smithi191 ceph-mon[33016]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:27.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:27 smithi053 ceph-mon[29182]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:27.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:27 smithi077 ceph-mon[32553]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:28.611 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.615+0000 7f813ffff700 1 -- 172.21.15.191:0/3038092703 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8158063860 con 0x7f81400a3c40 2024-04-28T12:46:28.611 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 8 on host 'smithi191' 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f81400a3c40 msgr2=0x7f81400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f81400a3c40 0x7f81400a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f814800b720 tx=0x7f8148005fd0 comp rx=0 tx=0).stop 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158104680 msgr2=0x7f8158199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158104680 0x7f8158199b90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f814c00e9d0 tx=0x7f814c00ece0 comp rx=0 tx=0).stop 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 shutdown_connections 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f81400a3c40 0x7f81400a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8158103480 0x7f8158199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8158105020 0x7f81581a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:28.613 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 --2- 172.21.15.191:0/3038092703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8158104680 0x7f8158199b90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:28.614 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.617+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 >> 172.21.15.191:0/3038092703 conn(0x7f81580fea20 msgr2=0x7f8158107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:28.614 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.618+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 shutdown_connections 2024-04-28T12:46:28.614 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:46:28.618+0000 7f815d615700 1 -- 172.21.15.191:0/3038092703 wait complete. 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:28.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:28.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:29.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:29.241 DEBUG:teuthology.orchestra.run.smithi191:osd.8> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.8.service 2024-04-28T12:46:29.243 INFO:tasks.cephadm:Deploying osd.9 on smithi053 with /dev/vg_nvme/lv_1... 2024-04-28T12:46:29.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T12:46:29.501 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:46:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:29 smithi191 ceph-mon[33016]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:29 smithi191 ceph-mon[33016]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T12:46:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:29 smithi191 ceph-mon[33016]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T12:46:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:29 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:29 smithi191 ceph-mon[33016]: from='osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:46:29.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:29 smithi191 ceph-mon[33016]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:46:29.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:29 smithi053 ceph-mon[29182]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:29.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:29 smithi053 ceph-mon[29182]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T12:46:29.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:29 smithi053 ceph-mon[29182]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T12:46:29.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:29 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:29.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:29 smithi053 ceph-mon[29182]: from='osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:46:29.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:29 smithi053 ceph-mon[29182]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:46:29.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:29 smithi077 ceph-mon[32553]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:29.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:29 smithi077 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T12:46:29.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:29 smithi077 ceph-mon[32553]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T12:46:29.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:29 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:29.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:29 smithi077 ceph-mon[32553]: from='osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:46:29.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:29 smithi077 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:46:30.395 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 12:46:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T12:46:30.093+0000 7fadd6dc6700 -1 osd.8 0 waiting for initial osdmap 2024-04-28T12:46:30.395 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 12:46:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T12:46:30.102+0000 7fadcebb1700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:46:30.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:30 smithi077 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:46:30.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:30 smithi077 ceph-mon[32553]: osdmap e59: 9 total, 8 up, 9 in 2024-04-28T12:46:30.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:30.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:30.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:30 smithi077 ceph-mon[32553]: from='osd.8 ' entity='osd.8' 2024-04-28T12:46:30.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:30 smithi053 ceph-mon[29182]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:46:30.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:30 smithi053 ceph-mon[29182]: osdmap e59: 9 total, 8 up, 9 in 2024-04-28T12:46:30.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:30.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:30.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:30 smithi053 ceph-mon[29182]: from='osd.8 ' entity='osd.8' 2024-04-28T12:46:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:30 smithi191 ceph-mon[33016]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:46:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:30 smithi191 ceph-mon[33016]: osdmap e59: 9 total, 8 up, 9 in 2024-04-28T12:46:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:30 smithi191 ceph-mon[33016]: from='osd.8 ' entity='osd.8' 2024-04-28T12:46:31.693 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T12:46:31.693 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T12:46:31.693 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-28T12:46:31.694 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-28T12:46:31.694 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00868146 s, 1.2 GB/s 2024-04-28T12:46:31.694 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-28T12:46:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:31 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:46:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:31 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:46:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:31 smithi191 ceph-mon[33016]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:31 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:31 smithi191 ceph-mon[33016]: osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] boot 2024-04-28T12:46:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:31 smithi191 ceph-mon[33016]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T12:46:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:31 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:31 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:46:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:31 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:46:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:31 smithi053 ceph-mon[29182]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:31 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:31 smithi053 ceph-mon[29182]: osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] boot 2024-04-28T12:46:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:31 smithi053 ceph-mon[29182]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T12:46:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:31 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:31.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:31 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:46:31.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:31 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:46:31.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:31 smithi077 ceph-mon[32553]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:31 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:31 smithi077 ceph-mon[32553]: osd.8 [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] boot 2024-04-28T12:46:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:31 smithi077 ceph-mon[32553]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T12:46:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:31 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:32.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_1 2024-04-28T12:46:32.773 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi191 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi191 to 5696M 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:32.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:32.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:32 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi191 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi191 to 5696M 2024-04-28T12:46:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:32 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi191 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi191 to 5696M 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:33.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:33.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:32 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:33 smithi053 ceph-mon[29182]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:33 smithi053 ceph-mon[29182]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T12:46:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:33 smithi191 ceph-mon[33016]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:33 smithi191 ceph-mon[33016]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T12:46:34.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:33 smithi077 ceph-mon[32553]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:33 smithi077 ceph-mon[32553]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T12:46:34.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 -- 172.21.15.53:0/2803073758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618104680 msgr2=0x7fc618104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:34.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 --2- 172.21.15.53:0/2803073758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618104680 0x7fc618104ae0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc6080099c0 tx=0x7fc608009cd0 comp rx=0 tx=0).stop 2024-04-28T12:46:34.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 -- 172.21.15.53:0/2803073758 shutdown_connections 2024-04-28T12:46:34.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 --2- 172.21.15.53:0/2803073758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc618105020 0x7fc61810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:34.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 --2- 172.21.15.53:0/2803073758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc618103480 0x7fc6181038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:34.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 --2- 172.21.15.53:0/2803073758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618104680 0x7fc618104ae0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:34.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 -- 172.21.15.53:0/2803073758 >> 172.21.15.53:0/2803073758 conn(0x7fc6180fea20 msgr2=0x7fc618100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:34.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 -- 172.21.15.53:0/2803073758 shutdown_connections 2024-04-28T12:46:34.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 -- 172.21.15.53:0/2803073758 wait complete. 2024-04-28T12:46:34.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 Processor -- start 2024-04-28T12:46:34.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.526+0000 7fc62065b700 1 -- start start 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc62065b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc618103480 0x7fc61819b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc62065b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc618104680 0x7fc61819bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc62065b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618105020 0x7fc6181a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc62065b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810ef50 con 0x7fc618105020 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc62065b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810edd0 con 0x7fc618104680 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc62065b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810f0d0 con 0x7fc618103480 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618105020 0x7fc6181a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618105020 0x7fc6181a2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42908/0 (socket says 172.21.15.53:42908) 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 -- 172.21.15.53:0/688695458 learned_addr learned my addr 172.21.15.53:0/688695458 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61dbf6700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc618104680 0x7fc61819bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61e3f7700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc618103480 0x7fc61819b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 -- 172.21.15.53:0/688695458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc618103480 msgr2=0x7fc61819b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:34.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc618103480 0x7fc61819b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 -- 172.21.15.53:0/688695458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc618104680 msgr2=0x7fc61819bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc618104680 0x7fc61819bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 -- 172.21.15.53:0/688695458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6181a3490 con 0x7fc618105020 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.527+0000 7fc61ebf8700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618105020 0x7fc6181a2df0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc610007f10 tx=0x7fc61000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.528+0000 7fc60f7fe700 1 -- 172.21.15.53:0/688695458 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610004bb0 con 0x7fc618105020 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.528+0000 7fc60f7fe700 1 -- 172.21.15.53:0/688695458 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc610004d50 con 0x7fc618105020 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.528+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6181a3780 con 0x7fc618105020 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.528+0000 7fc60f7fe700 1 -- 172.21.15.53:0/688695458 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100058e0 con 0x7fc618105020 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.528+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6181a3c70 con 0x7fc618105020 2024-04-28T12:46:34.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.529+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61806a110 con 0x7fc618105020 2024-04-28T12:46:34.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.530+0000 7fc60f7fe700 1 -- 172.21.15.53:0/688695458 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc61001a020 con 0x7fc618105020 2024-04-28T12:46:34.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.531+0000 7fc60f7fe700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc6040a3c40 0x7fc6040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:34.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.531+0000 7fc61e3f7700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc6040a3c40 0x7fc6040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:34.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.531+0000 7fc60f7fe700 1 -- 172.21.15.53:0/688695458 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fc6100cf800 con 0x7fc618105020 2024-04-28T12:46:34.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.531+0000 7fc61e3f7700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc6040a3c40 0x7fc6040a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc6181044e0 tx=0x7fc614007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:34.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.535+0000 7fc60f7fe700 1 -- 172.21.15.53:0/688695458 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc610010070 con 0x7fc618105020 2024-04-28T12:46:34.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:34.758+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc61810b570 con 0x7fc6040a3c40 2024-04-28T12:46:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:34 smithi191 ceph-mon[33016]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T12:46:35.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:34 smithi053 ceph-mon[29182]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T12:46:35.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:34 smithi077 ceph-mon[32553]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T12:46:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:35 smithi191 ceph-mon[33016]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:35 smithi191 ceph-mon[33016]: osdmap e64: 9 total, 9 up, 9 in 2024-04-28T12:46:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:35 smithi191 ceph-mon[33016]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:36.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:35 smithi053 ceph-mon[29182]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:36.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:35 smithi053 ceph-mon[29182]: osdmap e64: 9 total, 9 up, 9 in 2024-04-28T12:46:36.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:35 smithi053 ceph-mon[29182]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:36.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:36.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:36.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:36.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:35 smithi077 ceph-mon[32553]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:35 smithi077 ceph-mon[32553]: osdmap e64: 9 total, 9 up, 9 in 2024-04-28T12:46:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:35 smithi077 ceph-mon[32553]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:37 smithi191 ceph-mon[33016]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:37 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2033680821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aec50a3e-2db8-4933-812d-3ac318ddccdf"}]: dispatch 2024-04-28T12:46:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:37 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2033680821' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aec50a3e-2db8-4933-812d-3ac318ddccdf"}]': finished 2024-04-28T12:46:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:37 smithi191 ceph-mon[33016]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T12:46:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:37 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:38.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:37 smithi053 ceph-mon[29182]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:38.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:37 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2033680821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aec50a3e-2db8-4933-812d-3ac318ddccdf"}]: dispatch 2024-04-28T12:46:38.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:37 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2033680821' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aec50a3e-2db8-4933-812d-3ac318ddccdf"}]': finished 2024-04-28T12:46:38.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:37 smithi053 ceph-mon[29182]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T12:46:38.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:37 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:38.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:37 smithi077 ceph-mon[32553]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:38.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:37 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2033680821' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aec50a3e-2db8-4933-812d-3ac318ddccdf"}]: dispatch 2024-04-28T12:46:38.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:37 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2033680821' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aec50a3e-2db8-4933-812d-3ac318ddccdf"}]': finished 2024-04-28T12:46:38.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:37 smithi077 ceph-mon[32553]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T12:46:38.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:37 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:39.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:38 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3724923360' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:39.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:38 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3724923360' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:39.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:38 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3724923360' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:39 smithi191 ceph-mon[33016]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-28T12:46:40.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:39 smithi053 ceph-mon[29182]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-28T12:46:40.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:39 smithi077 ceph-mon[32553]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-28T12:46:41.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:41 smithi053 ceph-mon[29182]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-28T12:46:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:41 smithi191 ceph-mon[33016]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-28T12:46:42.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:41 smithi077 ceph-mon[32553]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-28T12:46:44.025 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:43 smithi053 ceph-mon[29182]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:46:44.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:43 smithi191 ceph-mon[33016]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:46:44.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:43 smithi077 ceph-mon[32553]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:46:45.998 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:45 smithi053 ceph-mon[29182]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-28T12:46:45.998 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T12:46:45.998 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:45.998 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:45 smithi053 ceph-mon[29182]: Deploying daemon osd.9 on smithi053 2024-04-28T12:46:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:45 smithi191 ceph-mon[33016]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-28T12:46:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T12:46:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:46.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:45 smithi191 ceph-mon[33016]: Deploying daemon osd.9 on smithi053 2024-04-28T12:46:46.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:45 smithi077 ceph-mon[32553]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-28T12:46:46.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T12:46:46.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:46.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:45 smithi077 ceph-mon[32553]: Deploying daemon osd.9 on smithi053 2024-04-28T12:46:48.000 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:47 smithi077 ceph-mon[32553]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T12:46:48.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:47 smithi191 ceph-mon[33016]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T12:46:48.056 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:47 smithi053 ceph-mon[29182]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T12:46:49.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:49 smithi053 ceph-mon[29182]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:50.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:49 smithi191 ceph-mon[33016]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:50.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:49 smithi077 ceph-mon[32553]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:52.020 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:51 smithi053 ceph-mon[29182]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:52.021 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:52.021 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:52.021 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:52.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:51 smithi191 ceph-mon[33016]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:52.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:51 smithi077 ceph-mon[32553]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:52.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:52.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:52.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:53 smithi191 ceph-mon[33016]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:54.091 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:53 smithi053 ceph-mon[29182]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:54.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:53 smithi077 ceph-mon[32553]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:55.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.369+0000 7fc60f7fe700 1 -- 172.21.15.53:0/688695458 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc61810b570 con 0x7fc6040a3c40 2024-04-28T12:46:55.365 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 9 on host 'smithi053' 2024-04-28T12:46:55.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc6040a3c40 msgr2=0x7fc6040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:55.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc6040a3c40 0x7fc6040a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc6181044e0 tx=0x7fc614007400 comp rx=0 tx=0).stop 2024-04-28T12:46:55.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618105020 msgr2=0x7fc6181a2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618105020 0x7fc6181a2df0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc610007f10 tx=0x7fc61000d3b0 comp rx=0 tx=0).stop 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 shutdown_connections 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc6040a3c40 0x7fc6040a6100 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc618105020 0x7fc6181a2df0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc618103480 0x7fc61819b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 --2- 172.21.15.53:0/688695458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc618104680 0x7fc61819bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 >> 172.21.15.53:0/688695458 conn(0x7fc6180fea20 msgr2=0x7fc618109e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 shutdown_connections 2024-04-28T12:46:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:46:55.371+0000 7fc62065b700 1 -- 172.21.15.53:0/688695458 wait complete. 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:55 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:55.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:55 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:56.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T12:46:56.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:56.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:46:56.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:55 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:56.149 DEBUG:teuthology.orchestra.run.smithi053:osd.9> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.9.service 2024-04-28T12:46:56.152 INFO:tasks.cephadm:Deploying osd.10 on smithi077 with /dev/vg_nvme/lv_1... 2024-04-28T12:46:56.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T12:46:56.399 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:46:56.779 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:56 smithi077 ceph-mon[32553]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T12:46:56.779 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:56 smithi077 ceph-mon[32553]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T12:46:56.779 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:56 smithi077 ceph-mon[32553]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:46:56.779 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:56 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:57.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:56 smithi191 ceph-mon[33016]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T12:46:57.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:56 smithi191 ceph-mon[33016]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T12:46:57.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:56 smithi191 ceph-mon[33016]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:46:57.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:56 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:57.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:56 smithi053 ceph-mon[29182]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T12:46:57.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:56 smithi053 ceph-mon[29182]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T12:46:57.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:56 smithi053 ceph-mon[29182]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-28T12:46:57.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:56 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:57.395 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T12:46:57.203+0000 7fd20d503700 -1 osd.9 0 waiting for initial osdmap 2024-04-28T12:46:57.395 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T12:46:57.215+0000 7fd204aed700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:46:57.859 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-mon[29182]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:57.859 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-mon[29182]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:46:57.859 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-mon[29182]: osdmap e67: 10 total, 9 up, 10 in 2024-04-28T12:46:57.859 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:57.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:57.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:57 smithi053 ceph-mon[29182]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' 2024-04-28T12:46:57.904 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:57 smithi077 ceph-mon[32553]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:57.904 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:57 smithi077 ceph-mon[32553]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:46:57.904 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:57 smithi077 ceph-mon[32553]: osdmap e67: 10 total, 9 up, 10 in 2024-04-28T12:46:57.904 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:57.904 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:57 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:57.904 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:57 smithi077 ceph-mon[32553]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' 2024-04-28T12:46:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:57 smithi191 ceph-mon[33016]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:57 smithi191 ceph-mon[33016]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-28T12:46:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:57 smithi191 ceph-mon[33016]: osdmap e67: 10 total, 9 up, 10 in 2024-04-28T12:46:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:57 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:57 smithi191 ceph-mon[33016]: from='osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776]' entity='osd.9' 2024-04-28T12:46:58.486 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T12:46:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T12:46:58.487 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-28T12:46:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-28T12:46:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131412 s, 798 MB/s 2024-04-28T12:46:58.487 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 2024-04-28T12:46:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-28T12:46:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:58 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:46:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:58 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:46:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:58 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:58 smithi191 ceph-mon[33016]: osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] boot 2024-04-28T12:46:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:58 smithi191 ceph-mon[33016]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T12:46:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:46:58 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:59.082 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:58 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:46:59.082 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:58 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:46:59.082 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:58 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:59.082 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:58 smithi053 ceph-mon[29182]: osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] boot 2024-04-28T12:46:59.082 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:58 smithi053 ceph-mon[29182]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T12:46:59.082 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:46:58 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:59.158 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:58 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:46:59.158 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:58 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:46:59.158 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:58 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:59.158 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:58 smithi077 ceph-mon[32553]: osd.9 [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] boot 2024-04-28T12:46:59.158 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:58 smithi077 ceph-mon[32553]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T12:46:59.158 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:46:58 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:59.185 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-04-28T12:46:59.424 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:00.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:00 smithi191 ceph-mon[33016]: osdmap e69: 10 total, 10 up, 10 in 2024-04-28T12:47:00.615 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:47:00.615 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.616 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:00 smithi077 ceph-mon[32553]: osdmap e69: 10 total, 10 up, 10 in 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:00.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:00.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:00 smithi053 ceph-mon[29182]: osdmap e69: 10 total, 10 up, 10 in 2024-04-28T12:47:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 -- 172.21.15.77:0/1327040116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c103f20 msgr2=0x7f9e1c10a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 --2- 172.21.15.77:0/1327040116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c103f20 0x7f9e1c10a7d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b0d0 tx=0x7f9e1400b3e0 comp rx=0 tx=0).stop 2024-04-28T12:47:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 -- 172.21.15.77:0/1327040116 shutdown_connections 2024-04-28T12:47:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 --2- 172.21.15.77:0/1327040116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c103f20 0x7f9e1c10a7d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 --2- 172.21.15.77:0/1327040116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e1c102380 0x7f9e1c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 --2- 172.21.15.77:0/1327040116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e1c103580 0x7f9e1c1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 -- 172.21.15.77:0/1327040116 >> 172.21.15.77:0/1327040116 conn(0x7f9e1c0fd920 msgr2=0x7f9e1c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 -- 172.21.15.77:0/1327040116 shutdown_connections 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.109+0000 7f9e23871700 1 -- 172.21.15.77:0/1327040116 wait complete. 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 Processor -- start 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 -- start start 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c102380 0x7f9e1c06bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e1c103580 0x7f9e1c19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e1c103f20 0x7f9e1c1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c10de10 con 0x7f9e1c103580 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c10dc90 con 0x7f9e1c102380 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e23871700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c10df90 con 0x7f9e1c103f20 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c102380 0x7f9e1c06bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c102380 0x7f9e1c06bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42462/0 (socket says 172.21.15.77:42462) 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 -- 172.21.15.77:0/2314350126 learned_addr learned my addr 172.21.15.77:0/2314350126 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-28T12:47:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e20e0c700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e1c103580 0x7f9e1c19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 -- 172.21.15.77:0/2314350126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e1c103f20 msgr2=0x7f9e1c1a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e1c103f20 0x7f9e1c1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 -- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e1c103580 msgr2=0x7f9e1c19e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e1c103580 0x7f9e1c19e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 -- 172.21.15.77:0/2314350126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c1a11f0 con 0x7f9e1c102380 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.110+0000 7f9e2160d700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c102380 0x7f9e1c06bc60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800c8f0 tx=0x7f9e1800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.111+0000 7f9e127fc700 1 -- 172.21.15.77:0/2314350126 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18004220 con 0x7f9e1c102380 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.111+0000 7f9e127fc700 1 -- 172.21.15.77:0/2314350126 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e18007db0 con 0x7f9e1c102380 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.111+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1c1a14e0 con 0x7f9e1c102380 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.111+0000 7f9e127fc700 1 -- 172.21.15.77:0/2314350126 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18003d10 con 0x7f9e1c102380 2024-04-28T12:47:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.111+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1c1a1a20 con 0x7f9e1c102380 2024-04-28T12:47:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.112+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1c06a110 con 0x7f9e1c102380 2024-04-28T12:47:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.113+0000 7f9e127fc700 1 -- 172.21.15.77:0/2314350126 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9e18007900 con 0x7f9e1c102380 2024-04-28T12:47:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.113+0000 7f9e127fc700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e080a3cc0 0x7f9e080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.114+0000 7f9e127fc700 1 -- 172.21.15.77:0/2314350126 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f9e180cf810 con 0x7f9e1c102380 2024-04-28T12:47:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.114+0000 7f9e20e0c700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e080a3cc0 0x7f9e080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:01.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.114+0000 7f9e20e0c700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e080a3cc0 0x7f9e080a6180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c005fd0 tx=0x7f9e0c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.118+0000 7f9e127fc700 1 -- 172.21.15.77:0/2314350126 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9e18014030 con 0x7f9e1c102380 2024-04-28T12:47:01.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:01.324+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9e1c063860 con 0x7f9e080a3cc0 2024-04-28T12:47:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:01 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi053 2024-04-28T12:47:01.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:01 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi053 to 4273M 2024-04-28T12:47:01.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:01 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi053 2024-04-28T12:47:01.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:01 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi053 to 4273M 2024-04-28T12:47:01.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:01 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi053 2024-04-28T12:47:01.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:01 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi053 to 4273M 2024-04-28T12:47:02.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:02 smithi077 ceph-mon[32553]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:02.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:02.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:02.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:02 smithi191 ceph-mon[33016]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:02.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:02.546 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:02 smithi053 ceph-mon[29182]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:02.546 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:02.546 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:02.546 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:03.363 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:03 smithi077 ceph-mon[32553]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:03 smithi191 ceph-mon[33016]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:03.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:03 smithi053 ceph-mon[29182]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:04.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:04 smithi191 ceph-mon[33016]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:04.590 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:04 smithi077 ceph-mon[32553]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:04.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:04 smithi053 ceph-mon[29182]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:05 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/859503963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]: dispatch 2024-04-28T12:47:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:05 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]: dispatch 2024-04-28T12:47:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:05 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]': finished 2024-04-28T12:47:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:05 smithi191 ceph-mon[33016]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T12:47:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:05 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:05.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:05 smithi191 ceph-mon[33016]: from='client.? 172.21.15.77:0/3119168826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:05 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/859503963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]: dispatch 2024-04-28T12:47:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:05 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]: dispatch 2024-04-28T12:47:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:05 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]': finished 2024-04-28T12:47:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:05 smithi053 ceph-mon[29182]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T12:47:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:05 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:05 smithi053 ceph-mon[29182]: from='client.? 172.21.15.77:0/3119168826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:05.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:05 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/859503963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]: dispatch 2024-04-28T12:47:05.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:05 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]: dispatch 2024-04-28T12:47:05.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:05 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1817077-d7a5-4eeb-8e2a-b948147d22a3"}]': finished 2024-04-28T12:47:05.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:05 smithi077 ceph-mon[32553]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T12:47:05.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:05 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:05.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:05 smithi077 ceph-mon[32553]: from='client.? 172.21.15.77:0/3119168826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:06 smithi191 ceph-mon[33016]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:06.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:06 smithi053 ceph-mon[29182]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:06.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:06 smithi077 ceph-mon[32553]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:08 smithi191 ceph-mon[33016]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:08.544 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:08 smithi077 ceph-mon[32553]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:08.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:08 smithi053 ceph-mon[29182]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:10.529 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:10 smithi077 ceph-mon[32553]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:10 smithi191 ceph-mon[33016]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:10.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:10 smithi053 ceph-mon[29182]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:12.508 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:12 smithi077 ceph-mon[32553]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:12.508 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T12:47:12.508 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:12.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:12 smithi191 ceph-mon[33016]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T12:47:12.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:12 smithi053 ceph-mon[29182]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T12:47:12.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:13.425 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:13 smithi077 ceph-mon[32553]: Deploying daemon osd.10 on smithi077 2024-04-28T12:47:13.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:13 smithi191 ceph-mon[33016]: Deploying daemon osd.10 on smithi077 2024-04-28T12:47:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:13 smithi053 ceph-mon[29182]: Deploying daemon osd.10 on smithi077 2024-04-28T12:47:14.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:14 smithi077 ceph-mon[32553]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:14 smithi191 ceph-mon[33016]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:14.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:14 smithi053 ceph-mon[29182]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:16.410 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:16 smithi077 ceph-mon[32553]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:16.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:16 smithi191 ceph-mon[33016]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:16.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:16 smithi053 ceph-mon[29182]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:18.382 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:18 smithi077 ceph-mon[32553]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:18.382 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:18.382 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:18.382 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:18 smithi191 ceph-mon[33016]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:18.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:18.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:18 smithi053 ceph-mon[29182]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:18.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:18.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:18.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:20.518 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:20 smithi077 ceph-mon[32553]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:20.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:20 smithi191 ceph-mon[33016]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:20 smithi053 ceph-mon[29182]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:21.743 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 10 on host 'smithi077' 2024-04-28T12:47:21.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.746+0000 7f9e127fc700 1 -- 172.21.15.77:0/2314350126 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9e1c063860 con 0x7f9e080a3cc0 2024-04-28T12:47:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e080a3cc0 msgr2=0x7f9e080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e080a3cc0 0x7f9e080a6180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c005fd0 tx=0x7f9e0c005f60 comp rx=0 tx=0).stop 2024-04-28T12:47:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c102380 msgr2=0x7f9e1c06bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c102380 0x7f9e1c06bc60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800c8f0 tx=0x7f9e1800ccb0 comp rx=0 tx=0).stop 2024-04-28T12:47:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 shutdown_connections 2024-04-28T12:47:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e080a3cc0 0x7f9e080a6180 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e1c103f20 0x7f9e1c1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c102380 0x7f9e1c06bc60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 --2- 172.21.15.77:0/2314350126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e1c103580 0x7f9e1c19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 >> 172.21.15.77:0/2314350126 conn(0x7f9e1c0fd920 msgr2=0x7f9e1c106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 shutdown_connections 2024-04-28T12:47:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-28T12:47:21.749+0000 7f9e23871700 1 -- 172.21.15.77:0/2314350126 wait complete. 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:21.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:21 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:21 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:21 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:22.395 DEBUG:teuthology.orchestra.run.smithi077:osd.10> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.10.service 2024-04-28T12:47:22.397 INFO:tasks.cephadm:Deploying osd.11 on smithi191 with /dev/vg_nvme/lv_1... 2024-04-28T12:47:22.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T12:47:22.644 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:47:23.279 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:23 smithi077 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T12:47:23.280 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:23 smithi077 ceph-mon[32553]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T12:47:23.280 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:23 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:23.280 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:23 smithi077 ceph-mon[32553]: from='osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:47:23.280 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:23 smithi077 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:23 smithi191 ceph-mon[33016]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T12:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:23 smithi191 ceph-mon[33016]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T12:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:23 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:23 smithi191 ceph-mon[33016]: from='osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:47:23.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:23 smithi191 ceph-mon[33016]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:47:23.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:23 smithi053 ceph-mon[29182]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T12:47:23.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:23 smithi053 ceph-mon[29182]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T12:47:23.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:23 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:23.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:23 smithi053 ceph-mon[29182]: from='osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:47:23.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:23 smithi053 ceph-mon[29182]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-28T12:47:23.948 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 12:47:23 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T12:47:23.696+0000 7f72adfc8700 -1 osd.10 0 waiting for initial osdmap 2024-04-28T12:47:23.948 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 12:47:23 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T12:47:23.709+0000 7f72a5db3700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:47:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:24 smithi191 ceph-mon[33016]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:24 smithi191 ceph-mon[33016]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:47:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:24 smithi191 ceph-mon[33016]: osdmap e72: 11 total, 10 up, 11 in 2024-04-28T12:47:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:24 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:24 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:24 smithi191 ceph-mon[33016]: from='osd.10 ' entity='osd.10' 2024-04-28T12:47:24.540 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:24 smithi077 ceph-mon[32553]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:24.540 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:24 smithi077 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:47:24.540 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:24 smithi077 ceph-mon[32553]: osdmap e72: 11 total, 10 up, 11 in 2024-04-28T12:47:24.540 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:24 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:24.541 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:24 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:24.541 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:24 smithi077 ceph-mon[32553]: from='osd.10 ' entity='osd.10' 2024-04-28T12:47:24.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:24 smithi053 ceph-mon[29182]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:24.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:24 smithi053 ceph-mon[29182]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-28T12:47:24.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:24 smithi053 ceph-mon[29182]: osdmap e72: 11 total, 10 up, 11 in 2024-04-28T12:47:24.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:24 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:24.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:24 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:24.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:24 smithi053 ceph-mon[29182]: from='osd.10 ' entity='osd.10' 2024-04-28T12:47:24.752 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T12:47:24.752 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T12:47:24.752 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-28T12:47:24.752 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-28T12:47:24.752 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112221 s, 934 MB/s 2024-04-28T12:47:24.752 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-28T12:47:25.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:25 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:47:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:25 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:47:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:25 smithi077 ceph-mon[32553]: osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] boot 2024-04-28T12:47:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:25 smithi077 ceph-mon[32553]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T12:47:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:25 smithi077 ceph-mon[32553]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T12:47:25.510 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:25 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:47:25.510 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:25 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:47:25.510 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:25 smithi191 ceph-mon[33016]: osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] boot 2024-04-28T12:47:25.510 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:25 smithi191 ceph-mon[33016]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T12:47:25.510 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:25.510 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:25 smithi191 ceph-mon[33016]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T12:47:25.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2024-04-28T12:47:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:25 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:47:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:25 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:47:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:25 smithi053 ceph-mon[29182]: osd.10 [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] boot 2024-04-28T12:47:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:25 smithi053 ceph-mon[29182]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T12:47:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:25 smithi053 ceph-mon[29182]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T12:47:25.778 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:47:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T12:47:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:26.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:26 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:26.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:26 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:26.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:26 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:27 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi077 2024-04-28T12:47:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:27 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi077 to 4272M 2024-04-28T12:47:27.311 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:27 smithi191 ceph-mon[33016]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 -- 172.21.15.191:0/2546104118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041046b0 msgr2=0x7fa404104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 --2- 172.21.15.191:0/2546104118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041046b0 0x7fa404104b10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009990 tx=0x7fa3ec009ca0 comp rx=0 tx=0).stop 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 -- 172.21.15.191:0/2546104118 shutdown_connections 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 --2- 172.21.15.191:0/2546104118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4041034b0 0x7fa4041038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 --2- 172.21.15.191:0/2546104118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa404105050 0x7fa40410b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 --2- 172.21.15.191:0/2546104118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041046b0 0x7fa404104b10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 -- 172.21.15.191:0/2546104118 >> 172.21.15.191:0/2546104118 conn(0x7fa4040fea70 msgr2=0x7fa404100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 -- 172.21.15.191:0/2546104118 shutdown_connections 2024-04-28T12:47:27.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.419+0000 7fa408f03700 1 -- 172.21.15.191:0/2546104118 wait complete. 2024-04-28T12:47:27.417 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 Processor -- start 2024-04-28T12:47:27.417 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 -- start start 2024-04-28T12:47:27.417 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041034b0 0x7fa4041995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:27.417 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4041046b0 0x7fa404199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa404105050 0x7fa4041a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa40410ef30 con 0x7fa404105050 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa40410edb0 con 0x7fa4041046b0 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa408f03700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa40410f0b0 con 0x7fa4041034b0 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa40259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041034b0 0x7fa4041995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa40259c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041034b0 0x7fa4041995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:55076/0 (socket says 172.21.15.191:55076) 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa40259c700 1 -- 172.21.15.191:0/336456857 learned_addr learned my addr 172.21.15.191:0/336456857 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa3fbfff700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4041046b0 0x7fa404199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.420+0000 7fa402d9d700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa404105050 0x7fa4041a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa40259c700 1 -- 172.21.15.191:0/336456857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4041046b0 msgr2=0x7fa404199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa40259c700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4041046b0 0x7fa404199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa40259c700 1 -- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa404105050 msgr2=0x7fa4041a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:27.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa40259c700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa404105050 0x7fa4041a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa40259c700 1 -- 172.21.15.191:0/336456857 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4041a1200 con 0x7fa4041034b0 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa40259c700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041034b0 0x7fa4041995a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400d9b0 tx=0x7fa3f400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa3fb7fe700 1 -- 172.21.15.191:0/336456857 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4017030 con 0x7fa4041034b0 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa3fb7fe700 1 -- 172.21.15.191:0/336456857 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3f40044d0 con 0x7fa4041034b0 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa3fb7fe700 1 -- 172.21.15.191:0/336456857 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40039e0 con 0x7fa4041034b0 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4041a14f0 con 0x7fa4041034b0 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.421+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4041a1a80 con 0x7fa4041034b0 2024-04-28T12:47:27.419 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.422+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa40406a110 con 0x7fa4041034b0 2024-04-28T12:47:27.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.423+0000 7fa3fb7fe700 1 -- 172.21.15.191:0/336456857 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa3f401a070 con 0x7fa4041034b0 2024-04-28T12:47:27.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.424+0000 7fa3fb7fe700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa3f00a3cc0 0x7fa3f00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:27.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.424+0000 7fa3fb7fe700 1 -- 172.21.15.191:0/336456857 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7fa3f40cfa30 con 0x7fa4041034b0 2024-04-28T12:47:27.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.424+0000 7fa3fbfff700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa3f00a3cc0 0x7fa3f00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:27.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.425+0000 7fa3fbfff700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa3f00a3cc0 0x7fa3f00a6180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00b5c0 tx=0x7fa3ec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:27.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.427+0000 7fa3fb7fe700 1 -- 172.21.15.191:0/336456857 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa3f400e350 con 0x7fa4041034b0 2024-04-28T12:47:27.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:27.633+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa404063860 con 0x7fa3f00a3cc0 2024-04-28T12:47:27.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:27 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi077 2024-04-28T12:47:27.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:27 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi077 to 4272M 2024-04-28T12:47:27.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:27 smithi053 ceph-mon[29182]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T12:47:27.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:27 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi077 2024-04-28T12:47:27.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:27 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi077 to 4272M 2024-04-28T12:47:27.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:27 smithi077 ceph-mon[32553]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T12:47:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:28 smithi191 ceph-mon[33016]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:28 smithi191 ceph-mon[33016]: osdmap e77: 11 total, 11 up, 11 in 2024-04-28T12:47:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:28 smithi191 ceph-mon[33016]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:28 smithi053 ceph-mon[29182]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:28 smithi053 ceph-mon[29182]: osdmap e77: 11 total, 11 up, 11 in 2024-04-28T12:47:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:28 smithi053 ceph-mon[29182]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:28.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:28 smithi077 ceph-mon[32553]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:28 smithi077 ceph-mon[32553]: osdmap e77: 11 total, 11 up, 11 in 2024-04-28T12:47:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:28 smithi077 ceph-mon[32553]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:29 smithi191 ceph-mon[33016]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 372 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:47:29.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:29 smithi053 ceph-mon[29182]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 372 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:47:29.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:29 smithi077 ceph-mon[32553]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 372 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:47:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:31 smithi191 ceph-mon[33016]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:47:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:31 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/3710582686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]: dispatch 2024-04-28T12:47:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:31 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]: dispatch 2024-04-28T12:47:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:31 smithi191 ceph-mon[33016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]': finished 2024-04-28T12:47:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:31 smithi191 ceph-mon[33016]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T12:47:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:31 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:31 smithi191 ceph-mon[33016]: from='client.? 172.21.15.191:0/1551398524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:31 smithi053 ceph-mon[29182]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:47:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:31 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/3710582686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]: dispatch 2024-04-28T12:47:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:31 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]: dispatch 2024-04-28T12:47:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:31 smithi053 ceph-mon[29182]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]': finished 2024-04-28T12:47:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:31 smithi053 ceph-mon[29182]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T12:47:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:31 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:31 smithi053 ceph-mon[29182]: from='client.? 172.21.15.191:0/1551398524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:31 smithi077 ceph-mon[32553]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:47:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:31 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/3710582686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]: dispatch 2024-04-28T12:47:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:31 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]: dispatch 2024-04-28T12:47:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:31 smithi077 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c0e510f-d109-48e1-844c-1e2146a8a412"}]': finished 2024-04-28T12:47:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:31 smithi077 ceph-mon[32553]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T12:47:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:31 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:31 smithi077 ceph-mon[32553]: from='client.? 172.21.15.191:0/1551398524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:33.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:33 smithi191 ceph-mon[33016]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:47:33.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:33 smithi053 ceph-mon[29182]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:47:33.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:33 smithi077 ceph-mon[32553]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:47:35.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:35 smithi191 ceph-mon[33016]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:47:35.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:35 smithi053 ceph-mon[29182]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:47:35.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:35 smithi077 ceph-mon[32553]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:47:37.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:37 smithi191 ceph-mon[33016]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:37.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:37 smithi053 ceph-mon[29182]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:37.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:37 smithi077 ceph-mon[32553]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:38 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T12:47:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:38 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:38 smithi191 ceph-mon[33016]: Deploying daemon osd.11 on smithi191 2024-04-28T12:47:38.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:38 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T12:47:38.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:38 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:38.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:38 smithi053 ceph-mon[29182]: Deploying daemon osd.11 on smithi191 2024-04-28T12:47:38.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:38 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T12:47:38.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:38 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:38.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:38 smithi077 ceph-mon[32553]: Deploying daemon osd.11 on smithi191 2024-04-28T12:47:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:39 smithi191 ceph-mon[33016]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:39.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:39 smithi053 ceph-mon[29182]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:39.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:39 smithi077 ceph-mon[32553]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:41.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:41 smithi191 ceph-mon[33016]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:41.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:41 smithi053 ceph-mon[29182]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:41.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:41 smithi077 ceph-mon[32553]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:43.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:43 smithi191 ceph-mon[33016]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:43 smithi053 ceph-mon[29182]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:43.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:43 smithi077 ceph-mon[32553]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:45.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:45.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:45.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:45.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:45.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:45.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:45.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:46.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:46 smithi191 ceph-mon[33016]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:46.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:46 smithi077 ceph-mon[32553]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:46.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:46 smithi053 ceph-mon[29182]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:47.698 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 11 on host 'smithi191' 2024-04-28T12:47:47.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.701+0000 7fa3fb7fe700 1 -- 172.21.15.191:0/336456857 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fa404063860 con 0x7fa3f00a3cc0 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa3f00a3cc0 msgr2=0x7fa3f00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa3f00a3cc0 0x7fa3f00a6180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00b5c0 tx=0x7fa3ec005fb0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041034b0 msgr2=0x7fa4041995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041034b0 0x7fa4041995a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400d9b0 tx=0x7fa3f400dcc0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 shutdown_connections 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa3f00a3cc0 0x7fa3f00a6180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa404105050 0x7fa4041a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4041034b0 0x7fa4041995a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 --2- 172.21.15.191:0/336456857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4041046b0 0x7fa404199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 >> 172.21.15.191:0/336456857 conn(0x7fa4040fea70 msgr2=0x7fa404107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 shutdown_connections 2024-04-28T12:47:47.702 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:47:47.705+0000 7fa408f03700 1 -- 172.21.15.191:0/336456857 wait complete. 2024-04-28T12:47:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:48 smithi191 ceph-mon[33016]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:48 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:48 smithi191 ceph-mon[33016]: from='osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:48 smithi191 ceph-mon[33016]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:48.351 DEBUG:teuthology.orchestra.run.smithi191:osd.11> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.11.service 2024-04-28T12:47:48.353 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-28T12:47:48.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-28T12:47:48.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:48 smithi077 ceph-mon[32553]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:48.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:48.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:48 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:48.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:48 smithi077 ceph-mon[32553]: from='osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:48.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:48 smithi077 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:48.614 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:47:48.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:48 smithi053 ceph-mon[29182]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:48.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:48.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:48 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:48.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:48 smithi053 ceph-mon[29182]: from='osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:48.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:48 smithi053 ceph-mon[29182]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:49.678 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T12:47:49.666+0000 7f6c88f5b700 -1 osd.11 0 waiting for initial osdmap 2024-04-28T12:47:49.678 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T12:47:49.678+0000 7f6c84d4e700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:47:49.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:49.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:49.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:49.679 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T12:47:49.679 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T12:47:49.679 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:49.679 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: from='osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:47:49.679 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:49 smithi191 ceph-mon[33016]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:47:49.848 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:49.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:49.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:49.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T12:47:49.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T12:47:49.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:49.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: from='osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:47:49.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:49 smithi053 ceph-mon[29182]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: from='osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:47:49.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:49 smithi077 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-28T12:47:50.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/31437255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84102380 msgr2=0x7f4c841027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/31437255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84102380 0x7f4c841027a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4c740099c0 tx=0x7f4c74009cd0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/31437255 shutdown_connections 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/31437255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84102380 0x7f4c841027a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/31437255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84103f20 0x7f4c8410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/31437255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c84103580 0x7f4c841039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/31437255 >> 172.21.15.53:0/31437255 conn(0x7f4c840fd920 msgr2=0x7f4c840ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.409+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/31437255 shutdown_connections 2024-04-28T12:47:50.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.410+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/31437255 wait complete. 2024-04-28T12:47:50.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.410+0000 7f4c8aeb5700 1 Processor -- start 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.410+0000 7f4c8aeb5700 1 -- start start 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c8aeb5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84102380 0x7f4c8419b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c8aeb5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c84103580 0x7f4c8419be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c8aeb5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84103f20 0x7f4c841a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c8aeb5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8410de90 con 0x7f4c84103f20 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c8aeb5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8410dd10 con 0x7f4c84103580 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c8aeb5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8410e010 con 0x7f4c84102380 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84102380 0x7f4c8419b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c89452700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84103f20 0x7f4c841a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84102380 0x7f4c8419b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.53:37836/0 (socket says 172.21.15.53:37836) 2024-04-28T12:47:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 -- 172.21.15.53:0/1416416847 learned_addr learned my addr 172.21.15.53:0/1416416847 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c83fff700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c84103580 0x7f4c8419be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 -- 172.21.15.53:0/1416416847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c84103580 msgr2=0x7f4c8419be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c84103580 0x7f4c8419be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 -- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84103f20 msgr2=0x7f4c841a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84103f20 0x7f4c841a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 -- 172.21.15.53:0/1416416847 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c841a3530 con 0x7f4c84102380 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c89452700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84103f20 0x7f4c841a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.411+0000 7f4c88c51700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84102380 0x7f4c8419b920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009f50 tx=0x7f4c74005af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.412+0000 7f4c81ffb700 1 -- 172.21.15.53:0/1416416847 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74027070 con 0x7f4c84102380 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.412+0000 7f4c81ffb700 1 -- 172.21.15.53:0/1416416847 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c74005940 con 0x7f4c84102380 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.412+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c841a37c0 con 0x7f4c84102380 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.412+0000 7f4c81ffb700 1 -- 172.21.15.53:0/1416416847 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74022bf0 con 0x7f4c84102380 2024-04-28T12:47:50.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.412+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c841a3ca0 con 0x7f4c84102380 2024-04-28T12:47:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.413+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c8406a110 con 0x7f4c84102380 2024-04-28T12:47:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.414+0000 7f4c81ffb700 1 -- 172.21.15.53:0/1416416847 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4c7402a070 con 0x7f4c84102380 2024-04-28T12:47:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.415+0000 7f4c81ffb700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4c6c0a3c40 0x7f4c6c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.415+0000 7f4c83fff700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4c6c0a3c40 0x7f4c6c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.415+0000 7f4c81ffb700 1 -- 172.21.15.53:0/1416416847 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f4c740d11f0 con 0x7f4c84102380 2024-04-28T12:47:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.415+0000 7f4c83fff700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4c6c0a3c40 0x7f4c6c0a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009a10 tx=0x7f4c78009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:50.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.418+0000 7f4c81ffb700 1 -- 172.21.15.53:0/1416416847 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4c74031070 con 0x7f4c84102380 2024-04-28T12:47:50.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.629+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f4c841a4110 con 0x7f4c84102380 2024-04-28T12:47:50.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.629+0000 7f4c81ffb700 1 -- 172.21.15.53:0/1416416847 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f4c74013070 con 0x7f4c84102380 2024-04-28T12:47:50.625 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:47:50.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4c6c0a3c40 msgr2=0x7f4c6c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:50.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4c6c0a3c40 0x7f4c6c0a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009a10 tx=0x7f4c78009190 comp rx=0 tx=0).stop 2024-04-28T12:47:50.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84102380 msgr2=0x7f4c8419b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:50.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84102380 0x7f4c8419b920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009f50 tx=0x7f4c74005af0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 shutdown_connections 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4c6c0a3c40 0x7f4c6c0a6100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c84103f20 0x7f4c841a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c84102380 0x7f4c8419b920 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 --2- 172.21.15.53:0/1416416847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c84103580 0x7f4c8419be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.631+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 >> 172.21.15.53:0/1416416847 conn(0x7f4c840fd920 msgr2=0x7f4c8406b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.632+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 shutdown_connections 2024-04-28T12:47:50.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:50.632+0000 7f4c8aeb5700 1 -- 172.21.15.53:0/1416416847 wait complete. 2024-04-28T12:47:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:50 smithi191 ceph-mon[33016]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:47:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:50 smithi191 ceph-mon[33016]: osdmap e80: 12 total, 11 up, 12 in 2024-04-28T12:47:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:50 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:50 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1416416847' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:50.924 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:50 smithi053 ceph-mon[29182]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:47:50.924 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:50 smithi053 ceph-mon[29182]: osdmap e80: 12 total, 11 up, 12 in 2024-04-28T12:47:50.924 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:50.924 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:50 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:50.924 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:50 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1416416847' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:50.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:50 smithi077 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-28T12:47:50.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:50 smithi077 ceph-mon[32553]: osdmap e80: 12 total, 11 up, 12 in 2024-04-28T12:47:50.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:50.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:50 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:50 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1416416847' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:51.295 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1714308444,"num_in_osds":12,"osd_in_since":1714308450,"num_remapped_pgs":0} 2024-04-28T12:47:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:51 smithi077 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T12:47:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:51 smithi077 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T12:47:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:51 smithi077 ceph-mon[32553]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:51 smithi077 ceph-mon[32553]: osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] boot 2024-04-28T12:47:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:51 smithi077 ceph-mon[32553]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T12:47:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:51 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:51.967 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:51 smithi191 ceph-mon[33016]: purged_snaps scrub starts 2024-04-28T12:47:51.967 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:51 smithi191 ceph-mon[33016]: purged_snaps scrub ok 2024-04-28T12:47:51.967 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:51 smithi191 ceph-mon[33016]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:51.967 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:51 smithi191 ceph-mon[33016]: osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] boot 2024-04-28T12:47:51.968 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:51 smithi191 ceph-mon[33016]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T12:47:51.968 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:51 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:52.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:51 smithi053 ceph-mon[29182]: purged_snaps scrub starts 2024-04-28T12:47:52.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:51 smithi053 ceph-mon[29182]: purged_snaps scrub ok 2024-04-28T12:47:52.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:51 smithi053 ceph-mon[29182]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:52.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:51 smithi053 ceph-mon[29182]: osd.11 [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] boot 2024-04-28T12:47:52.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:51 smithi053 ceph-mon[29182]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T12:47:52.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:51 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:52.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-28T12:47:52.545 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: Detected new or changed devices on smithi191 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi191 to 4272M 2024-04-28T12:47:52.938 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.939 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:52.939 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:52.939 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.939 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:52 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: Detected new or changed devices on smithi191 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi191 to 4272M 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:52.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:52.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:52 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: Detected new or changed devices on smithi191 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi191 to 4272M 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:53.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:53.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:53.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:52 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:53.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:53 smithi077 ceph-mon[32553]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:53.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:53 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:53.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:53 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:53.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:53 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:53.965 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:53 smithi053 ceph-mon[29182]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:53.965 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:53 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:53.965 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:53 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:53.965 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:53 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:54.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:53 smithi191 ceph-mon[33016]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:53 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:53 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:53 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 -- 172.21.15.53:0/2670484214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08701023a0 msgr2=0x7f08701027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 --2- 172.21.15.53:0/2670484214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08701023a0 0x7f08701027c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f08640099c0 tx=0x7f0864009cd0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 -- 172.21.15.53:0/2670484214 shutdown_connections 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 --2- 172.21.15.53:0/2670484214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08701023a0 0x7f08701027c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 --2- 172.21.15.53:0/2670484214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0870103f40 0x7f087010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 --2- 172.21.15.53:0/2670484214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08701035a0 0x7f0870103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 -- 172.21.15.53:0/2670484214 >> 172.21.15.53:0/2670484214 conn(0x7f08700fd980 msgr2=0x7f08700ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 -- 172.21.15.53:0/2670484214 shutdown_connections 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.206+0000 7f0875a06700 1 -- 172.21.15.53:0/2670484214 wait complete. 2024-04-28T12:47:54.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 Processor -- start 2024-04-28T12:47:54.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 -- start start 2024-04-28T12:47:54.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08701023a0 0x7f08701984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08701035a0 0x7f08701989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0870103f40 0x7f087019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f087010de10 con 0x7f0870103f40 2024-04-28T12:47:54.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f087010dc90 con 0x7f08701035a0 2024-04-28T12:47:54.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.207+0000 7f0875a06700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f087010df90 con 0x7f08701023a0 2024-04-28T12:47:54.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0870103f40 0x7f087019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0870103f40 0x7f087019fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50820/0 (socket says 172.21.15.53:50820) 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 -- 172.21.15.53:0/3585018012 learned_addr learned my addr 172.21.15.53:0/3585018012 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086e7fc700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08701035a0 0x7f08701989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086effd700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08701023a0 0x7f08701984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 -- 172.21.15.53:0/3585018012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08701023a0 msgr2=0x7f08701984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08701023a0 0x7f08701984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 -- 172.21.15.53:0/3585018012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08701035a0 msgr2=0x7f08701989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08701035a0 0x7f08701989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 -- 172.21.15.53:0/3585018012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08701a0100 con 0x7f0870103f40 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f086f7fe700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0870103f40 0x7f087019fa60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0868007da0 tx=0x7f086800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f0874a04700 1 -- 172.21.15.53:0/3585018012 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868016020 con 0x7f0870103f40 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f0874a04700 1 -- 172.21.15.53:0/3585018012 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f086800ad00 con 0x7f0870103f40 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f087006bbf0 con 0x7f0870103f40 2024-04-28T12:47:54.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f0874a04700 1 -- 172.21.15.53:0/3585018012 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08680115e0 con 0x7f0870103f40 2024-04-28T12:47:54.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.208+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f087006c130 con 0x7f0870103f40 2024-04-28T12:47:54.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.209+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f087006a110 con 0x7f0870103f40 2024-04-28T12:47:54.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.210+0000 7f0874a04700 1 -- 172.21.15.53:0/3585018012 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0868019020 con 0x7f0870103f40 2024-04-28T12:47:54.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.211+0000 7f0874a04700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f08580a3c40 0x7f08580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.211+0000 7f086effd700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f08580a3c40 0x7f08580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.211+0000 7f0874a04700 1 -- 172.21.15.53:0/3585018012 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f08680cff40 con 0x7f0870103f40 2024-04-28T12:47:54.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.212+0000 7f086effd700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f08580a3c40 0x7f08580a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0864003790 tx=0x7f0864005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:54.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.216+0000 7f0874a04700 1 -- 172.21.15.53:0/3585018012 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0868020020 con 0x7f0870103f40 2024-04-28T12:47:54.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.424+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0870109420 con 0x7f0870103f40 2024-04-28T12:47:54.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.424+0000 7f0874a04700 1 -- 172.21.15.53:0/3585018012 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f0868019310 con 0x7f0870103f40 2024-04-28T12:47:54.421 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.426+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f08580a3c40 msgr2=0x7f08580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.426+0000 7f0875a06700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f08580a3c40 0x7f08580a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0864003790 tx=0x7f0864005fb0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.426+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0870103f40 msgr2=0x7f087019fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.426+0000 7f0875a06700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0870103f40 0x7f087019fa60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0868007da0 tx=0x7f086800a340 comp rx=0 tx=0).stop 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 shutdown_connections 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f08580a3c40 0x7f08580a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0870103f40 0x7f087019fa60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08701023a0 0x7f08701984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 --2- 172.21.15.53:0/3585018012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08701035a0 0x7f08701989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 >> 172.21.15.53:0/3585018012 conn(0x7f08700fd980 msgr2=0x7f0870106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:54.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 shutdown_connections 2024-04-28T12:47:54.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:54.427+0000 7f0875a06700 1 -- 172.21.15.53:0/3585018012 wait complete. 2024-04-28T12:47:54.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:54 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3585018012' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:55.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:54 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3585018012' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:55.128 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:54 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3585018012' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:55.162 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1714308470,"num_in_osds":12,"osd_in_since":1714308450,"num_remapped_pgs":0} 2024-04-28T12:47:55.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T12:47:55.414 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:47:55.849 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:55 smithi053 ceph-mon[29182]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:55.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:55 smithi077 ceph-mon[32553]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:55 smithi191 ceph-mon[33016]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:57.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 -- 172.21.15.53:0/4157358537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090102320 msgr2=0x7fa0901027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 --2- 172.21.15.53:0/4157358537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090102320 0x7fa0901027a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa084009990 tx=0x7fa084009ca0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 -- 172.21.15.53:0/4157358537 shutdown_connections 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 --2- 172.21.15.53:0/4157358537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa090102ce0 0x7fa09010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 --2- 172.21.15.53:0/4157358537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa090108390 0x7fa090108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 --2- 172.21.15.53:0/4157358537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090102320 0x7fa0901027a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 -- 172.21.15.53:0/4157358537 >> 172.21.15.53:0/4157358537 conn(0x7fa0900792d0 msgr2=0x7fa0900796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 -- 172.21.15.53:0/4157358537 shutdown_connections 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.162+0000 7fa0956a8700 1 -- 172.21.15.53:0/4157358537 wait complete. 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 Processor -- start 2024-04-28T12:47:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 -- start start 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa090102320 0x7fa09019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa090102ce0 0x7fa09019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090108390 0x7fa090199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa090112530 con 0x7fa090108390 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0901123b0 con 0x7fa090102320 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa0956a8700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa0901126b0 con 0x7fa090102ce0 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090108390 0x7fa090199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa08f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090108390 0x7fa090199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49562/0 (socket says 172.21.15.53:49562) 2024-04-28T12:47:57.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa08f7fe700 1 -- 172.21.15.53:0/2838311401 learned_addr learned my addr 172.21.15.53:0/2838311401 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.163+0000 7fa08effd700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa090102320 0x7fa09019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08e7fc700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa090102ce0 0x7fa09019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08f7fe700 1 -- 172.21.15.53:0/2838311401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa090102ce0 msgr2=0x7fa09019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08f7fe700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa090102ce0 0x7fa09019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08f7fe700 1 -- 172.21.15.53:0/2838311401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa090102320 msgr2=0x7fa09019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08f7fe700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa090102320 0x7fa09019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08f7fe700 1 -- 172.21.15.53:0/2838311401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09019a080 con 0x7fa090108390 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08e7fc700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa090102ce0 0x7fa09019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa08f7fe700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090108390 0x7fa090199920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa088007ee0 tx=0x7fa08800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa077fff700 1 -- 172.21.15.53:0/2838311401 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08800dbd0 con 0x7fa090108390 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa077fff700 1 -- 172.21.15.53:0/2838311401 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa08800dd70 con 0x7fa090108390 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa09019a370 con 0x7fa090108390 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa077fff700 1 -- 172.21.15.53:0/2838311401 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa088005990 con 0x7fa090108390 2024-04-28T12:47:57.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.164+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0901a6630 con 0x7fa090108390 2024-04-28T12:47:57.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.165+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa09006a110 con 0x7fa090108390 2024-04-28T12:47:57.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.166+0000 7fa077fff700 1 -- 172.21.15.53:0/2838311401 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa08801a020 con 0x7fa090108390 2024-04-28T12:47:57.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.167+0000 7fa077fff700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0780a3c40 0x7fa0780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.167+0000 7fa08effd700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0780a3c40 0x7fa0780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.167+0000 7fa077fff700 1 -- 172.21.15.53:0/2838311401 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa0880cfd40 con 0x7fa090108390 2024-04-28T12:47:57.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.167+0000 7fa08effd700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0780a3c40 0x7fa0780a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa09019aa60 tx=0x7fa08000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:57.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.171+0000 7fa077fff700 1 -- 172.21.15.53:0/2838311401 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa088010070 con 0x7fa090108390 2024-04-28T12:47:57.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.379+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0901a6860 con 0x7fa090108390 2024-04-28T12:47:57.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.380+0000 7fa077fff700 1 -- 172.21.15.53:0/2838311401 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18421 (secure 0 0 0) 0x7fa08809cbc0 con 0x7fa090108390 2024-04-28T12:47:57.376 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:47:57.376 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:47:51.666968+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:47:57.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0780a3c40 msgr2=0x7fa0780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0780a3c40 0x7fa0780a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa09019aa60 tx=0x7fa08000b410 comp rx=0 tx=0).stop 2024-04-28T12:47:57.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090108390 msgr2=0x7fa090199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090108390 0x7fa090199920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa088007ee0 tx=0x7fa08800d3b0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 shutdown_connections 2024-04-28T12:47:57.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0780a3c40 0x7fa0780a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa090108390 0x7fa090199920 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa090102320 0x7fa09019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 --2- 172.21.15.53:0/2838311401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa090102ce0 0x7fa09019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 >> 172.21.15.53:0/2838311401 conn(0x7fa0900792d0 msgr2=0x7fa090101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:57.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 shutdown_connections 2024-04-28T12:47:57.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:47:57.382+0000 7fa0956a8700 1 -- 172.21.15.53:0/2838311401 wait complete. 2024-04-28T12:47:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:57 smithi077 ceph-mon[32553]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:57 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2838311401' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:58.016 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:57 smithi053 ceph-mon[29182]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:58.016 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:57 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2838311401' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:57 smithi191 ceph-mon[33016]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:57 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2838311401' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:58.046 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:47:58.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-28T12:47:58.293 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:47:59.807 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:47:59 smithi053 ceph-mon[29182]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:59.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:47:59 smithi077 ceph-mon[32553]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:47:59 smithi191 ceph-mon[33016]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 -- 172.21.15.53:0/2131091978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f406bad0 msgr2=0x7fe0f41102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2131091978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f406bad0 0x7fe0f41102e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00b0a0 tx=0x7fe0ec00b3b0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 -- 172.21.15.53:0/2131091978 shutdown_connections 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2131091978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f406bad0 0x7fe0f41102e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2131091978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f41094a0 0x7fe0f4109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2131091978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0f406b130 0x7fe0f406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 -- 172.21.15.53:0/2131091978 >> 172.21.15.53:0/2131091978 conn(0x7fe0f40792d0 msgr2=0x7fe0f40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:00.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 -- 172.21.15.53:0/2131091978 shutdown_connections 2024-04-28T12:48:00.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.068+0000 7fe0fb472700 1 -- 172.21.15.53:0/2131091978 wait complete. 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 Processor -- start 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 -- start start 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0f406b130 0x7fe0f419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f406bad0 0x7fe0f419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f41094a0 0x7fe0f4199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41135a0 con 0x7fe0f41094a0 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4113420 con 0x7fe0f406bad0 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.069+0000 7fe0fb472700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4113720 con 0x7fe0f406b130 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f41094a0 0x7fe0f4199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f41094a0 0x7fe0f4199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49588/0 (socket says 172.21.15.53:49588) 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 -- 172.21.15.53:0/2952153092 learned_addr learned my addr 172.21.15.53:0/2952153092 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f8a0d700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f406bad0 0x7fe0f419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f920e700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0f406b130 0x7fe0f419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 -- 172.21.15.53:0/2952153092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0f406b130 msgr2=0x7fe0f419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0f406b130 0x7fe0f419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 -- 172.21.15.53:0/2952153092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f406bad0 msgr2=0x7fe0f419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f406bad0 0x7fe0f419fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 -- 172.21.15.53:0/2952153092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f419a020 con 0x7fe0f41094a0 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f920e700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0f406b130 0x7fe0f419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0f9a0f700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f41094a0 0x7fe0f4199840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00b830 tx=0x7fe0ec004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0ea7fc700 1 -- 172.21.15.53:0/2952153092 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec021070 con 0x7fe0f41094a0 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0ea7fc700 1 -- 172.21.15.53:0/2952153092 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec007e30 con 0x7fe0f41094a0 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0ea7fc700 1 -- 172.21.15.53:0/2952153092 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec0047d0 con 0x7fe0f41094a0 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f419a2b0 con 0x7fe0f41094a0 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.070+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f41a6520 con 0x7fe0f41094a0 2024-04-28T12:48:00.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.071+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0f4104b50 con 0x7fe0f41094a0 2024-04-28T12:48:00.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.073+0000 7fe0ea7fc700 1 -- 172.21.15.53:0/2952153092 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe0ec007980 con 0x7fe0f41094a0 2024-04-28T12:48:00.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.073+0000 7fe0ea7fc700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe0e00a3cc0 0x7fe0e00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.073+0000 7fe0f920e700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe0e00a3cc0 0x7fe0e00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.073+0000 7fe0ea7fc700 1 -- 172.21.15.53:0/2952153092 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe0ec0d0fa0 con 0x7fe0f41094a0 2024-04-28T12:48:00.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.074+0000 7fe0f920e700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe0e00a3cc0 0x7fe0e00a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005e10 tx=0x7fe0f0005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:00.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.078+0000 7fe0ea7fc700 1 -- 172.21.15.53:0/2952153092 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe0ec09deb0 con 0x7fe0f41094a0 2024-04-28T12:48:00.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.283+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe0f419ad70 con 0x7fe0f41094a0 2024-04-28T12:48:00.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.283+0000 7fe0ea7fc700 1 -- 172.21.15.53:0/2952153092 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fe0ec09d7b0 con 0x7fe0f41094a0 2024-04-28T12:48:00.279 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T12:48:00.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe0e00a3cc0 msgr2=0x7fe0e00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe0e00a3cc0 0x7fe0e00a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0005e10 tx=0x7fe0f0005d60 comp rx=0 tx=0).stop 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f41094a0 msgr2=0x7fe0f4199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f41094a0 0x7fe0f4199840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00b830 tx=0x7fe0ec004ce0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 shutdown_connections 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe0e00a3cc0 0x7fe0e00a6180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0f41094a0 0x7fe0f4199840 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0f406b130 0x7fe0f419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 --2- 172.21.15.53:0/2952153092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f406bad0 0x7fe0f419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 >> 172.21.15.53:0/2952153092 conn(0x7fe0f40792d0 msgr2=0x7fe0f4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 shutdown_connections 2024-04-28T12:48:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:00.285+0000 7fe0fb472700 1 -- 172.21.15.53:0/2952153092 wait complete. 2024-04-28T12:48:00.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:00 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2952153092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:01.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:00 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2952153092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:01.042 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:00 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2952153092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:01.077 INFO:tasks.cephadm:Adding mds.a on smithi053 2024-04-28T12:48:01.077 INFO:tasks.cephadm:Adding mds.d on smithi053 2024-04-28T12:48:01.077 INFO:tasks.cephadm:Adding mds.g on smithi053 2024-04-28T12:48:01.077 INFO:tasks.cephadm:Adding mds.j on smithi053 2024-04-28T12:48:01.077 INFO:tasks.cephadm:Adding mds.b on smithi077 2024-04-28T12:48:01.077 INFO:tasks.cephadm:Adding mds.e on smithi077 2024-04-28T12:48:01.078 INFO:tasks.cephadm:Adding mds.h on smithi077 2024-04-28T12:48:01.078 INFO:tasks.cephadm:Adding mds.k on smithi077 2024-04-28T12:48:01.078 INFO:tasks.cephadm:Adding mds.c on smithi191 2024-04-28T12:48:01.078 INFO:tasks.cephadm:Adding mds.f on smithi191 2024-04-28T12:48:01.078 INFO:tasks.cephadm:Adding mds.i on smithi191 2024-04-28T12:48:01.078 INFO:tasks.cephadm:Adding mds.l on smithi191 2024-04-28T12:48:01.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l' 2024-04-28T12:48:01.328 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:48:01.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:01 smithi191 ceph-mon[33016]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:01 smithi077 ceph-mon[32553]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:01.968 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:01 smithi053 ceph-mon[29182]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/265818502 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0102370 msgr2=0x7f7ce0102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/265818502 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0102370 0x7f7ce0102790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009990 tx=0x7f7cd4009ca0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/265818502 shutdown_connections 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/265818502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce0103f10 0x7f7ce010a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/265818502 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0102370 0x7f7ce0102790 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/265818502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ce0103570 0x7f7ce01039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/265818502 >> 172.21.15.191:0/265818502 conn(0x7f7ce00fd910 msgr2=0x7f7ce00ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/265818502 shutdown_connections 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.128+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/265818502 wait complete. 2024-04-28T12:48:03.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 Processor -- start 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 -- start start 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ce0102370 0x7f7ce006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0103570 0x7f7ce019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce0103f10 0x7f7ce01a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce010dee0 con 0x7f7ce0102370 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce010dd60 con 0x7f7ce0103f10 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce6cfb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce010e060 con 0x7f7ce0103570 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7cdffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0103570 0x7f7ce019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7cdffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0103570 0x7f7ce019e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:34650/0 (socket says 172.21.15.191:34650) 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7cdffff700 1 -- 172.21.15.191:0/2824969975 learned_addr learned my addr 172.21.15.191:0/2824969975 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce4a97700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ce0102370 0x7f7ce006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.129+0000 7f7ce5298700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce0103f10 0x7f7ce01a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cdffff700 1 -- 172.21.15.191:0/2824969975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce0103f10 msgr2=0x7f7ce01a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:03.128 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cdffff700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce0103f10 0x7f7ce01a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cdffff700 1 -- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ce0102370 msgr2=0x7f7ce006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cdffff700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ce0102370 0x7f7ce006c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cdffff700 1 -- 172.21.15.191:0/2824969975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce01a1310 con 0x7f7ce0103570 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cdffff700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0103570 0x7f7ce019e450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00b530 tx=0x7f7ccc00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cddffb700 1 -- 172.21.15.191:0/2824969975 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc015020 con 0x7f7ce0103570 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cddffb700 1 -- 172.21.15.191:0/2824969975 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ccc011ce0 con 0x7f7ce0103570 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce01a1600 con 0x7f7ce0103570 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7cddffb700 1 -- 172.21.15.191:0/2824969975 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc010dd0 con 0x7f7ce0103570 2024-04-28T12:48:03.129 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.130+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce01a1bc0 con 0x7f7ce0103570 2024-04-28T12:48:03.130 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.131+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce006a110 con 0x7f7ce0103570 2024-04-28T12:48:03.130 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.132+0000 7f7cddffb700 1 -- 172.21.15.191:0/2824969975 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7ccc018020 con 0x7f7ce0103570 2024-04-28T12:48:03.130 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.133+0000 7f7cddffb700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7cd00a3c40 0x7f7cd00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:03.130 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.133+0000 7f7cddffb700 1 -- 172.21.15.191:0/2824969975 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7ccc0cf6c0 con 0x7f7ce0103570 2024-04-28T12:48:03.130 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.133+0000 7f7ce4a97700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7cd00a3c40 0x7f7cd00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:03.130 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.134+0000 7f7ce4a97700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7cd00a3c40 0x7f7cd00a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40038a0 tx=0x7f7cd4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:03.134 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.137+0000 7f7cddffb700 1 -- 172.21.15.191:0/2824969975 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7ccc01f020 con 0x7f7ce0103570 2024-04-28T12:48:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.355+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}) -- 0x7f7ce0063a60 con 0x7f7cd00a3c40 2024-04-28T12:48:03.359 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.363+0000 7f7cddffb700 1 -- 172.21.15.191:0/2824969975 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f7ce0063a60 con 0x7f7cd00a3c40 2024-04-28T12:48:03.360 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mds.all update... 2024-04-28T12:48:03.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7cd00a3c40 msgr2=0x7f7cd00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:03.362 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7cd00a3c40 0x7f7cd00a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40038a0 tx=0x7f7cd4005fb0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0103570 msgr2=0x7f7ce019e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0103570 0x7f7ce019e450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00b530 tx=0x7f7ccc00b840 comp rx=0 tx=0).stop 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 shutdown_connections 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7cd00a3c40 0x7f7cd00a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ce0102370 0x7f7ce006c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce0103f10 0x7f7ce01a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 --2- 172.21.15.191:0/2824969975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ce0103570 0x7f7ce019e450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 >> 172.21.15.191:0/2824969975 conn(0x7f7ce00fd910 msgr2=0x7f7ce0106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 shutdown_connections 2024-04-28T12:48:03.363 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-28T12:48:03.366+0000 7f7ce6cfb700 1 -- 172.21.15.191:0/2824969975 wait complete. 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:04.007 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:04.007 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:04.007 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:04.007 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:04.007 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:04.007 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:04.008 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:04.008 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:04.008 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:04.037 DEBUG:teuthology.orchestra.run.smithi053:mds.a> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.a.service 2024-04-28T12:48:04.040 DEBUG:teuthology.orchestra.run.smithi053:mds.d> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.d.service 2024-04-28T12:48:04.065 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:04.066 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:04.067 DEBUG:teuthology.orchestra.run.smithi053:mds.g> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.g.service 2024-04-28T12:48:04.109 DEBUG:teuthology.orchestra.run.smithi053:mds.j> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.j.service 2024-04-28T12:48:04.152 DEBUG:teuthology.orchestra.run.smithi077:mds.b> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.b.service 2024-04-28T12:48:04.155 DEBUG:teuthology.orchestra.run.smithi077:mds.e> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.e.service 2024-04-28T12:48:04.200 DEBUG:teuthology.orchestra.run.smithi077:mds.h> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.h.service 2024-04-28T12:48:04.242 DEBUG:teuthology.orchestra.run.smithi077:mds.k> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.k.service 2024-04-28T12:48:04.286 DEBUG:teuthology.orchestra.run.smithi191:mds.c> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.c.service 2024-04-28T12:48:04.288 DEBUG:teuthology.orchestra.run.smithi191:mds.f> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.f.service 2024-04-28T12:48:04.331 DEBUG:teuthology.orchestra.run.smithi191:mds.i> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.i.service 2024-04-28T12:48:04.373 DEBUG:teuthology.orchestra.run.smithi191:mds.l> sudo journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.l.service 2024-04-28T12:48:04.416 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-28T12:48:04.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:04.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 -- 172.21.15.53:0/4061850640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 msgr2=0x7f8dfc109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 --2- 172.21.15.53:0/4061850640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc109cb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009ca0 tx=0x7f8df402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 -- 172.21.15.53:0/4061850640 shutdown_connections 2024-04-28T12:48:04.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 --2- 172.21.15.53:0/4061850640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc109cb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 --2- 172.21.15.53:0/4061850640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc102a60 0x7f8dfc102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 --2- 172.21.15.53:0/4061850640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dfc101870 0x7f8dfc101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 -- 172.21.15.53:0/4061850640 >> 172.21.15.53:0/4061850640 conn(0x7f8dfc0fcde0 msgr2=0x7f8dfc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 -- 172.21.15.53:0/4061850640 shutdown_connections 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.530+0000 7f8e017c9640 1 -- 172.21.15.53:0/4061850640 wait complete. 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 Processor -- start 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 -- start start 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc101870 0x7f8dfc198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dfc102a60 0x7f8dfc1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc1a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc10d380 con 0x7f8dfc103420 2024-04-28T12:48:04.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc10d200 con 0x7f8dfc102a60 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8e017c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc10d500 con 0x7f8dfc101870 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc1a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc1a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49612/0 (socket says 172.21.15.53:49612) 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8dfb7fe640 1 -- 172.21.15.53:0/3174062563 learned_addr learned my addr 172.21.15.53:0/3174062563 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8dfa7fc640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dfc102a60 0x7f8dfc1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.531+0000 7f8dfaffd640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc101870 0x7f8dfc198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8dfb7fe640 1 -- 172.21.15.53:0/3174062563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc101870 msgr2=0x7f8dfc198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8dfb7fe640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc101870 0x7f8dfc198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8dfb7fe640 1 -- 172.21.15.53:0/3174062563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dfc102a60 msgr2=0x7f8dfc1990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8dfb7fe640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dfc102a60 0x7f8dfc1990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8dfb7fe640 1 -- 172.21.15.53:0/3174062563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc1a0790 con 0x7f8dfc103420 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8dfb7fe640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc1a0120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8df40097b0 tx=0x7f8df4031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8de3fff640 1 -- 172.21.15.53:0/3174062563 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df4003bf0 con 0x7f8dfc103420 2024-04-28T12:48:04.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dfc1a0a20 con 0x7f8dfc103420 2024-04-28T12:48:04.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8de3fff640 1 -- 172.21.15.53:0/3174062563 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8df4003d90 con 0x7f8dfc103420 2024-04-28T12:48:04.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8de3fff640 1 -- 172.21.15.53:0/3174062563 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df402fc70 con 0x7f8dfc103420 2024-04-28T12:48:04.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.532+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dfc1a0fc0 con 0x7f8dfc103420 2024-04-28T12:48:04.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.533+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dfc1a1250 con 0x7f8dfc103420 2024-04-28T12:48:04.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.534+0000 7f8de3fff640 1 -- 172.21.15.53:0/3174062563 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8df4004040 con 0x7f8dfc103420 2024-04-28T12:48:04.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.535+0000 7f8de3fff640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8de40a3bb0 0x7f8de40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.535+0000 7f8dfaffd640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8de40a3bb0 0x7f8de40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.535+0000 7f8de3fff640 1 -- 172.21.15.53:0/3174062563 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8df40f14f0 con 0x7f8dfc103420 2024-04-28T12:48:04.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.536+0000 7f8dfaffd640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8de40a3bb0 0x7f8de40a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc1028c0 tx=0x7f8dec00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:04.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.540+0000 7f8de3fff640 1 -- 172.21.15.53:0/3174062563 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8df40be370 con 0x7f8dfc103420 2024-04-28T12:48:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.731+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8dfc1a1850 con 0x7f8dfc103420 2024-04-28T12:48:04.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.732+0000 7f8de3fff640 1 -- 172.21.15.53:0/3174062563 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18421 (secure 0 0 0) 0x7f8df40bdc70 con 0x7f8dfc103420 2024-04-28T12:48:04.728 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:04.728 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:47:51.666968+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8de40a3bb0 msgr2=0x7f8de40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8de40a3bb0 0x7f8de40a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc1028c0 tx=0x7f8dec00a2d0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 msgr2=0x7f8dfc1a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc1a0120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8df40097b0 tx=0x7f8df4031840 comp rx=0 tx=0).stop 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 shutdown_connections 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8de40a3bb0 0x7f8de40a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dfc103420 0x7f8dfc1a0120 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dfc102a60 0x7f8dfc1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 --2- 172.21.15.53:0/3174062563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc101870 0x7f8dfc198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 >> 172.21.15.53:0/3174062563 conn(0x7f8dfc0fcde0 msgr2=0x7f8dfc0fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 shutdown_connections 2024-04-28T12:48:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.735+0000 7f8e017c9640 1 -- 172.21.15.53:0/3174062563 wait complete. 2024-04-28T12:48:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:48:04.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:48:04.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:04 smithi191 ceph-mon[33016]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:04 smithi191 ceph-mon[33016]: Saving service mds.all spec with placement smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-28T12:48:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:04 smithi191 ceph-mon[33016]: Deploying daemon mds.l on smithi191 2024-04-28T12:48:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 -- 172.21.15.53:0/1567687331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 msgr2=0x7f42bc102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/1567687331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc102ee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009900 tx=0x7f42b002a410 comp rx=0 tx=0).stop 2024-04-28T12:48:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 -- 172.21.15.53:0/1567687331 shutdown_connections 2024-04-28T12:48:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/1567687331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc103420 0x7f42bc109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/1567687331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc102ee0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/1567687331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42bc101870 0x7f42bc101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 -- 172.21.15.53:0/1567687331 >> 172.21.15.53:0/1567687331 conn(0x7f42bc0fcde0 msgr2=0x7f42bc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 -- 172.21.15.53:0/1567687331 shutdown_connections 2024-04-28T12:48:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.852+0000 7f42c0ecc640 1 -- 172.21.15.53:0/1567687331 wait complete. 2024-04-28T12:48:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 Processor -- start 2024-04-28T12:48:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 -- start start 2024-04-28T12:48:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc101870 0x7f42bc10c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc10a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42bc103420 0x7f42bc10ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc10e4c0 con 0x7f42bc102a60 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc10e340 con 0x7f42bc101870 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42c0ecc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc10e640 con 0x7f42bc103420 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc10a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc10a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49632/0 (socket says 172.21.15.53:49632) 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 -- 172.21.15.53:0/3236788014 learned_addr learned my addr 172.21.15.53:0/3236788014 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42bad9d640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42bc103420 0x7f42bc10ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42ba59c640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc101870 0x7f42bc10c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 -- 172.21.15.53:0/3236788014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42bc103420 msgr2=0x7f42bc10ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42bc103420 0x7f42bc10ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 -- 172.21.15.53:0/3236788014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc101870 msgr2=0x7f42bc10c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc101870 0x7f42bc10c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.853+0000 7f42b9d9b640 1 -- 172.21.15.53:0/3236788014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc10b600 con 0x7f42bc102a60 2024-04-28T12:48:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.854+0000 7f42b9d9b640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc10a830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f42b0005940 tx=0x7f42b0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.854+0000 7f42ab7fe640 1 -- 172.21.15.53:0/3236788014 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b003f030 con 0x7f42bc102a60 2024-04-28T12:48:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.854+0000 7f42ab7fe640 1 -- 172.21.15.53:0/3236788014 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42b0003cb0 con 0x7f42bc102a60 2024-04-28T12:48:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.854+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc1a2ca0 con 0x7f42bc102a60 2024-04-28T12:48:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.854+0000 7f42ab7fe640 1 -- 172.21.15.53:0/3236788014 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0035070 con 0x7f42bc102a60 2024-04-28T12:48:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.854+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc1a3190 con 0x7f42bc102a60 2024-04-28T12:48:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.855+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42bc108d20 con 0x7f42bc102a60 2024-04-28T12:48:04.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.856+0000 7f42ab7fe640 1 -- 172.21.15.53:0/3236788014 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f42b0042030 con 0x7f42bc102a60 2024-04-28T12:48:04.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.857+0000 7f42ab7fe640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42a40a3c30 0x7f42a40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:04.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.857+0000 7f42ba59c640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42a40a3c30 0x7f42a40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:04.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.857+0000 7f42ab7fe640 1 -- 172.21.15.53:0/3236788014 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f42b00f09a0 con 0x7f42bc102a60 2024-04-28T12:48:04.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.858+0000 7f42ba59c640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42a40a3c30 0x7f42a40a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f42ac005fd0 tx=0x7f42ac005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:04.862+0000 7f42ab7fe640 1 -- 172.21.15.53:0/3236788014 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f42b0003e60 con 0x7f42bc102a60 2024-04-28T12:48:05.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.049+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f42bc063530 con 0x7f42bc102a60 2024-04-28T12:48:05.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.049+0000 7f42ab7fe640 1 -- 172.21.15.53:0/3236788014 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f42b00bd780 con 0x7f42bc102a60 2024-04-28T12:48:05.045 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T12:48:05.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42a40a3c30 msgr2=0x7f42a40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42a40a3c30 0x7f42a40a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f42ac005fd0 tx=0x7f42ac005850 comp rx=0 tx=0).stop 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 msgr2=0x7f42bc10a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc10a830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f42b0005940 tx=0x7f42b0004430 comp rx=0 tx=0).stop 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 shutdown_connections 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42a40a3c30 0x7f42a40a60f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42bc103420 0x7f42bc10ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42bc102a60 0x7f42bc10a830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 --2- 172.21.15.53:0/3236788014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42bc101870 0x7f42bc10c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 >> 172.21.15.53:0/3236788014 conn(0x7f42bc0fcde0 msgr2=0x7f42bc105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 shutdown_connections 2024-04-28T12:48:05.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.051+0000 7f42c0ecc640 1 -- 172.21.15.53:0/3236788014 wait complete. 2024-04-28T12:48:05.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:04 smithi053 ceph-mon[29182]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:05.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:04 smithi053 ceph-mon[29182]: Saving service mds.all spec with placement smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-28T12:48:05.057 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:04 smithi053 ceph-mon[29182]: Deploying daemon mds.l on smithi191 2024-04-28T12:48:05.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:05.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 -- 172.21.15.53:0/4039067230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101bc0 msgr2=0x7fd1c0102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 --2- 172.21.15.53:0/4039067230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101bc0 0x7fd1c0102020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009900 tx=0x7fd1ac02a480 comp rx=0 tx=0).stop 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 -- 172.21.15.53:0/4039067230 shutdown_connections 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 --2- 172.21.15.53:0/4039067230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0102560 0x7fd1c01029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 --2- 172.21.15.53:0/4039067230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101bc0 0x7fd1c0102020 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 --2- 172.21.15.53:0/4039067230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1c0101260 0x7fd1c0101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 -- 172.21.15.53:0/4039067230 >> 172.21.15.53:0/4039067230 conn(0x7fd1c00fcde0 msgr2=0x7fd1c00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 -- 172.21.15.53:0/4039067230 shutdown_connections 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.164+0000 7fd1c7103640 1 -- 172.21.15.53:0/4039067230 wait complete. 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 Processor -- start 2024-04-28T12:48:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 -- start start 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101260 0x7fd1c019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1c0101bc0 0x7fd1c019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0102560 0x7fd1c01a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c01106f0 con 0x7fd1c0101260 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c0110570 con 0x7fd1c0102560 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c7103640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c0110870 con 0x7fd1c0101bc0 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c4e9f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101260 0x7fd1c019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c4e9f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101260 0x7fd1c019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49658/0 (socket says 172.21.15.53:49658) 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c4e9f640 1 -- 172.21.15.53:0/912110209 learned_addr learned my addr 172.21.15.53:0/912110209 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1c56a0640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0102560 0x7fd1c01a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.165+0000 7fd1bffff640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1c0101bc0 0x7fd1c019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c4e9f640 1 -- 172.21.15.53:0/912110209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1c0101bc0 msgr2=0x7fd1c019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c4e9f640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1c0101bc0 0x7fd1c019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c4e9f640 1 -- 172.21.15.53:0/912110209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0102560 msgr2=0x7fd1c01a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c4e9f640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0102560 0x7fd1c01a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c4e9f640 1 -- 172.21.15.53:0/912110209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c01a29e0 con 0x7fd1c0101260 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c56a0640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0102560 0x7fd1c01a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1bffff640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1c0101bc0 0x7fd1c019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c4e9f640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101260 0x7fd1c019adc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e9f0 tx=0x7fd1b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1bdffb640 1 -- 172.21.15.53:0/912110209 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b400cd80 con 0x7fd1c0101260 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c01a2cd0 con 0x7fd1c0101260 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1bdffb640 1 -- 172.21.15.53:0/912110209 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b40054d0 con 0x7fd1c0101260 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1bdffb640 1 -- 172.21.15.53:0/912110209 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4010690 con 0x7fd1c0101260 2024-04-28T12:48:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.166+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c01a31d0 con 0x7fd1c0101260 2024-04-28T12:48:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.167+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c010c1b0 con 0x7fd1c0101260 2024-04-28T12:48:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.168+0000 7fd1bdffb640 1 -- 172.21.15.53:0/912110209 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd1b4004e90 con 0x7fd1c0101260 2024-04-28T12:48:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.169+0000 7fd1bdffb640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd1b00a3bb0 0x7fd1b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.169+0000 7fd1bffff640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd1b00a3bb0 0x7fd1b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.169+0000 7fd1bdffb640 1 -- 172.21.15.53:0/912110209 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd1b40d0550 con 0x7fd1c0101260 2024-04-28T12:48:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.169+0000 7fd1bffff640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd1b00a3bb0 0x7fd1b00a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac02a9a0 tx=0x7fd1ac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:05.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.173+0000 7fd1bdffb640 1 -- 172.21.15.53:0/912110209 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd1b409d330 con 0x7fd1c0101260 2024-04-28T12:48:05.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:04 smithi077 ceph-mon[32553]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:05.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:04 smithi077 ceph-mon[32553]: Saving service mds.all spec with placement smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-28T12:48:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:04 smithi077 ceph-mon[32553]: Deploying daemon mds.l on smithi191 2024-04-28T12:48:05.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.398+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1c0063560 con 0x7fd1c0101260 2024-04-28T12:48:05.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.398+0000 7fd1bdffb640 1 -- 172.21.15.53:0/912110209 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fd1b409cc30 con 0x7fd1c0101260 2024-04-28T12:48:05.394 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:05.394 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-28T12:48:05.395 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 1 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd1b00a3bb0 msgr2=0x7fd1b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd1b00a3bb0 0x7fd1b00a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac02a9a0 tx=0x7fd1ac005fd0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101260 msgr2=0x7fd1c019adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101260 0x7fd1c019adc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e9f0 tx=0x7fd1b400edf0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 shutdown_connections 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd1b00a3bb0 0x7fd1b00a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c0102560 0x7fd1c01a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1c0101bc0 0x7fd1c019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 --2- 172.21.15.53:0/912110209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1c0101260 0x7fd1c019adc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 >> 172.21.15.53:0/912110209 conn(0x7fd1c00fcde0 msgr2=0x7fd1c00fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 shutdown_connections 2024-04-28T12:48:05.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:05.401+0000 7fd1c7103640 1 -- 172.21.15.53:0/912110209 wait complete. 2024-04-28T12:48:06.000 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:05 smithi191 ceph-mon[33016]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:06.000 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:05 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3174062563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:06.000 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:05 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3236788014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:06.000 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:05 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/912110209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:06.000 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:05 smithi191 systemd[1]: Starting Ceph mds.l for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:06.001 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 podman[63236]: 2024-04-28 12:48:05.907179992 +0000 UTC m=+0.018203589 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:05 smithi053 ceph-mon[29182]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:05 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3174062563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:05 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3236788014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:06.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:05 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/912110209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:06.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:05 smithi077 ceph-mon[32553]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:06.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:05 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3174062563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:06.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:05 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3236788014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:06.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:05 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/912110209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:06.269 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 podman[63236]: 2024-04-28 12:48:06.036798557 +0000 UTC m=+0.147822129 container create 6196d3c0dcf5604f34c63b93ffb5d17f959b91e0678458d14a63de6903d06e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-28T12:48:06.537 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 podman[63236]: 2024-04-28 12:48:06.274807239 +0000 UTC m=+0.385830818 container init 6196d3c0dcf5604f34c63b93ffb5d17f959b91e0678458d14a63de6903d06e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-28T12:48:06.537 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 podman[63236]: 2024-04-28 12:48:06.277599507 +0000 UTC m=+0.388623089 container start 6196d3c0dcf5604f34c63b93ffb5d17f959b91e0678458d14a63de6903d06e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-28T12:48:06.537 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 ceph-mds[63255]: -- 172.21.15.191:0/1948304526 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561cdeca1a40 con 0x561cdec75000 2024-04-28T12:48:06.537 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l[63251]: starting mds.l at 2024-04-28T12:48:06.537 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 bash[63236]: 6196d3c0dcf5604f34c63b93ffb5d17f959b91e0678458d14a63de6903d06e43 2024-04-28T12:48:06.537 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 12:48:06 smithi191 systemd[1]: Started Ceph mds.l for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:07.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:07.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6832/1697828385,v1:172.21.15.191:6833/1697828385] up:boot 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: fsmap 1 up:standby 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:07.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:07 smithi077 ceph-mon[32553]: Deploying daemon mds.k on smithi077 2024-04-28T12:48:07.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- 172.21.15.53:0/2552682092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 msgr2=0x7f4afc102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 --2- 172.21.15.53:0/2552682092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc102ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009900 tx=0x7f4aec02a410 comp rx=0 tx=0).stop 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- 172.21.15.53:0/2552682092 shutdown_connections 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 --2- 172.21.15.53:0/2552682092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4afc103420 0x7f4afc109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 --2- 172.21.15.53:0/2552682092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc102ee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 --2- 172.21.15.53:0/2552682092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4afc101870 0x7f4afc101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- 172.21.15.53:0/2552682092 >> 172.21.15.53:0/2552682092 conn(0x7f4afc0fcde0 msgr2=0x7f4afc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- 172.21.15.53:0/2552682092 shutdown_connections 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- 172.21.15.53:0/2552682092 wait complete. 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 Processor -- start 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- start start 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4afc101870 0x7f4afc197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4afc103420 0x7f4afc19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc10d3d0 con 0x7f4afc102a60 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc10d250 con 0x7f4afc101870 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.529+0000 7f4b0375f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc10d550 con 0x7f4afc103420 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44848/0 (socket says 172.21.15.53:44848) 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 -- 172.21.15.53:0/543467161 learned_addr learned my addr 172.21.15.53:0/543467161 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b014fb640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4afc101870 0x7f4afc197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b01cfc640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4afc103420 0x7f4afc19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 -- 172.21.15.53:0/543467161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4afc103420 msgr2=0x7f4afc19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4afc103420 0x7f4afc19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 -- 172.21.15.53:0/543467161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4afc101870 msgr2=0x7f4afc197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4afc101870 0x7f4afc197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 -- 172.21.15.53:0/543467161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc19f720 con 0x7f4afc102a60 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b014fb640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4afc101870 0x7f4afc197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b01cfc640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4afc103420 0x7f4afc19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b00cfa640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc198040 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4aec005ab0 tx=0x7f4aec003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4af27fc640 1 -- 172.21.15.53:0/543467161 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec03f070 con 0x7f4afc102a60 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4af27fc640 1 -- 172.21.15.53:0/543467161 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aec031c50 con 0x7f4afc102a60 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc19f9b0 con 0x7f4afc102a60 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4af27fc640 1 -- 172.21.15.53:0/543467161 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec035030 con 0x7f4afc102a60 2024-04-28T12:48:07.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.530+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4afc19fea0 con 0x7f4afc102a60 2024-04-28T12:48:07.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.531+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4afc108bc0 con 0x7f4afc102a60 2024-04-28T12:48:07.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.532+0000 7f4af27fc640 1 -- 172.21.15.53:0/543467161 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4aec042070 con 0x7f4afc102a60 2024-04-28T12:48:07.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.533+0000 7f4af27fc640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ae80a3bb0 0x7f4ae80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.533+0000 7f4b014fb640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ae80a3bb0 0x7f4ae80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.533+0000 7f4af27fc640 1 -- 172.21.15.53:0/543467161 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4aec0f07e0 con 0x7f4afc102a60 2024-04-28T12:48:07.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.534+0000 7f4b014fb640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ae80a3bb0 0x7f4ae80a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005fd0 tx=0x7f4af800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:07.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.538+0000 7f4af27fc640 1 -- 172.21.15.53:0/543467161 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4aec0bd660 con 0x7f4afc102a60 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6832/1697828385,v1:172.21.15.191:6833/1697828385] up:boot 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: fsmap 1 up:standby 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:07.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:07.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:07.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:07 smithi053 ceph-mon[29182]: Deploying daemon mds.k on smithi077 2024-04-28T12:48:07.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.765+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4afc02dad0 con 0x7f4afc102a60 2024-04-28T12:48:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.766+0000 7f4af27fc640 1 -- 172.21.15.53:0/543467161 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f4aec0bcf60 con 0x7f4afc102a60 2024-04-28T12:48:07.762 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:07.762 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-28T12:48:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 2 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ae80a3bb0 msgr2=0x7f4ae80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ae80a3bb0 0x7f4ae80a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4af8005fd0 tx=0x7f4af800b410 comp rx=0 tx=0).stop 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 msgr2=0x7f4afc198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc198040 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4aec005ab0 tx=0x7f4aec003b90 comp rx=0 tx=0).stop 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 shutdown_connections 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ae80a3bb0 0x7f4ae80a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4afc103420 0x7f4afc19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4afc102a60 0x7f4afc198040 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 --2- 172.21.15.53:0/543467161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4afc101870 0x7f4afc197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 >> 172.21.15.53:0/543467161 conn(0x7f4afc0fcde0 msgr2=0x7f4afc105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:07.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 shutdown_connections 2024-04-28T12:48:07.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:07.768+0000 7f4b0375f640 1 -- 172.21.15.53:0/543467161 wait complete. 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6832/1697828385,v1:172.21.15.191:6833/1697828385] up:boot 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: fsmap 1 up:standby 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:07.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:07 smithi191 ceph-mon[33016]: Deploying daemon mds.k on smithi077 2024-04-28T12:48:08.559 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:08 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/543467161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:08.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:08 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/543467161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:08.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:08 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/543467161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:08.948 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:08 smithi077 systemd[1]: Starting Ceph mds.k for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:09.210 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:08 smithi077 podman[62085]: 2024-04-28 12:48:08.857231094 +0000 UTC m=+0.017884899 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:09.210 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:08 smithi077 podman[62085]: 2024-04-28 12:48:08.970708714 +0000 UTC m=+0.131362514 container create 231e8a50d16b4eab6a2dc09b47eedc558e598715e8123bf1e22f30f8a44b8ea5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-28T12:48:09.210 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:09 smithi077 podman[62085]: 2024-04-28 12:48:09.215527321 +0000 UTC m=+0.376181121 container init 231e8a50d16b4eab6a2dc09b47eedc558e598715e8123bf1e22f30f8a44b8ea5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True) 2024-04-28T12:48:09.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mon[32553]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:09.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:09.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:09.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:09.699 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:09 smithi077 podman[62085]: 2024-04-28 12:48:09.218165501 +0000 UTC m=+0.378819303 container start 231e8a50d16b4eab6a2dc09b47eedc558e598715e8123bf1e22f30f8a44b8ea5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-28T12:48:09.699 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-mds[62104]: -- 172.21.15.77:0/1688998164 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564da4413a40 con 0x564da43e6c00 2024-04-28T12:48:09.699 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:09 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k[62100]: starting mds.k at 2024-04-28T12:48:09.699 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:09 smithi077 bash[62085]: 231e8a50d16b4eab6a2dc09b47eedc558e598715e8123bf1e22f30f8a44b8ea5 2024-04-28T12:48:09.699 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 12:48:09 smithi077 systemd[1]: Started Ceph mds.k for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:09.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:09 smithi191 ceph-mon[33016]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:09.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:09.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:09 smithi053 ceph-mon[29182]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:09.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:09.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:09.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:09.797 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:09.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 -- 172.21.15.53:0/291505416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688103410 msgr2=0x7f7688105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 --2- 172.21.15.53:0/291505416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688103410 0x7f7688105800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f767c009900 tx=0x7f767c02a410 comp rx=0 tx=0).stop 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 -- 172.21.15.53:0/291505416 shutdown_connections 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 --2- 172.21.15.53:0/291505416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108670 0x7f768810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 --2- 172.21.15.53:0/291505416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688105d40 0x7f7688108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 --2- 172.21.15.53:0/291505416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688103410 0x7f7688105800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 -- 172.21.15.53:0/291505416 >> 172.21.15.53:0/291505416 conn(0x7f76880fce20 msgr2=0x7f76880ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 -- 172.21.15.53:0/291505416 shutdown_connections 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.884+0000 7f768e16a640 1 -- 172.21.15.53:0/291505416 wait complete. 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 Processor -- start 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 -- start start 2024-04-28T12:48:09.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688103410 0x7f7688102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688105d40 0x7f76881013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108670 0x7f7688101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f768810e4c0 con 0x7f7688105d40 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f768810e340 con 0x7f7688108670 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.885+0000 7f768e16a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f768810e640 con 0x7f7688103410 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688105d40 0x7f76881013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688105d40 0x7f76881013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44858/0 (socket says 172.21.15.53:44858) 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7687fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108670 0x7f7688101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 -- 172.21.15.53:0/2672855354 learned_addr learned my addr 172.21.15.53:0/2672855354 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f76877fe640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688103410 0x7f7688102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 -- 172.21.15.53:0/2672855354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688103410 msgr2=0x7f7688102d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688103410 0x7f7688102d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 -- 172.21.15.53:0/2672855354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108670 msgr2=0x7f7688101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:09.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108670 0x7f7688101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 -- 172.21.15.53:0/2672855354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7688102190 con 0x7f7688105d40 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7687fff640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108670 0x7f7688101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7686ffd640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688105d40 0x7f76881013c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f767800d770 tx=0x7f767800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7684ff9640 1 -- 172.21.15.53:0/2672855354 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678017070 con 0x7f7688105d40 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7684ff9640 1 -- 172.21.15.53:0/2672855354 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7678009c30 con 0x7f7688105d40 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76881a2d50 con 0x7f7688105d40 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f7684ff9640 1 -- 172.21.15.53:0/2672855354 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767800f620 con 0x7f7688105d40 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.886+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76881a31e0 con 0x7f7688105d40 2024-04-28T12:48:09.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.888+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76880fe910 con 0x7f7688105d40 2024-04-28T12:48:09.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.889+0000 7f7684ff9640 1 -- 172.21.15.53:0/2672855354 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f767801a070 con 0x7f7688105d40 2024-04-28T12:48:09.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.889+0000 7f7684ff9640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f76700a3bb0 0x7f76700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:09.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.889+0000 7f7684ff9640 1 -- 172.21.15.53:0/2672855354 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f76780cf9f0 con 0x7f7688105d40 2024-04-28T12:48:09.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.889+0000 7f76877fe640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f76700a3bb0 0x7f76700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:09.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.890+0000 7f76877fe640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f76700a3bb0 0x7f76700a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f767c000c00 tx=0x7f767c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:09.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:09.894+0000 7f7684ff9640 1 -- 172.21.15.53:0/2672855354 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7678009de0 con 0x7f7688105d40 2024-04-28T12:48:10.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.131+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f768802dad0 con 0x7f7688105d40 2024-04-28T12:48:10.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.131+0000 7f7684ff9640 1 -- 172.21.15.53:0/2672855354 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f767809c220 con 0x7f7688105d40 2024-04-28T12:48:10.130 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:10.130 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 3 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f76700a3bb0 msgr2=0x7f76700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f76700a3bb0 0x7f76700a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f767c000c00 tx=0x7f767c0314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688105d40 msgr2=0x7f76881013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688105d40 0x7f76881013c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f767800d770 tx=0x7f767800dc30 comp rx=0 tx=0).stop 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 shutdown_connections 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f76700a3bb0 0x7f76700a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108670 0x7f7688101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7688105d40 0x7f76881013c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 --2- 172.21.15.53:0/2672855354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7688103410 0x7f7688102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 >> 172.21.15.53:0/2672855354 conn(0x7f76880fce20 msgr2=0x7f7688103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:10.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 shutdown_connections 2024-04-28T12:48:10.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:10.133+0000 7f768e16a640 1 -- 172.21.15.53:0/2672855354 wait complete. 2024-04-28T12:48:10.621 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:10 smithi077 ceph-mon[32553]: Deploying daemon mds.j on smithi053 2024-04-28T12:48:10.621 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:10 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6832/3944921810,v1:172.21.15.77:6833/3944921810] up:boot 2024-04-28T12:48:10.622 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:10 smithi077 ceph-mon[32553]: fsmap 2 up:standby 2024-04-28T12:48:10.622 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T12:48:10.622 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:10 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2672855354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:10.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:10 smithi053 ceph-mon[29182]: Deploying daemon mds.j on smithi053 2024-04-28T12:48:10.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:10 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6832/3944921810,v1:172.21.15.77:6833/3944921810] up:boot 2024-04-28T12:48:10.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:10 smithi053 ceph-mon[29182]: fsmap 2 up:standby 2024-04-28T12:48:10.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T12:48:10.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:10 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2672855354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:10 smithi191 ceph-mon[33016]: Deploying daemon mds.j on smithi053 2024-04-28T12:48:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:10 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6832/3944921810,v1:172.21.15.77:6833/3944921810] up:boot 2024-04-28T12:48:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:10 smithi191 ceph-mon[33016]: fsmap 2 up:standby 2024-04-28T12:48:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T12:48:10.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:10 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2672855354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:11.576 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:11 smithi053 ceph-mon[29182]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:11.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:11 smithi077 ceph-mon[32553]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:11 smithi191 ceph-mon[33016]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:11.895 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:11 smithi053 systemd[1]: Starting Ceph mds.j for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:12.141 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:12.240 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:11 smithi053 podman[69915]: 2024-04-28 12:48:11.891245589 +0000 UTC m=+0.018202188 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:12.240 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:12 smithi053 podman[69915]: 2024-04-28 12:48:12.000462062 +0000 UTC m=+0.127418706 container create 534b2f156e3af79d1adaa3c7c3f2e494046be293838e5b7d8b399abfeef69bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T12:48:12.240 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:12 smithi053 podman[69915]: 2024-04-28 12:48:12.245257281 +0000 UTC m=+0.372213862 container init 534b2f156e3af79d1adaa3c7c3f2e494046be293838e5b7d8b399abfeef69bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2) 2024-04-28T12:48:12.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 -- 172.21.15.53:0/3316740021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0103420 msgr2=0x7f20e0109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 --2- 172.21.15.53:0/3316740021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0103420 0x7f20e0109cb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009ca0 tx=0x7f20d802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 -- 172.21.15.53:0/3316740021 shutdown_connections 2024-04-28T12:48:12.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 --2- 172.21.15.53:0/3316740021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0103420 0x7f20e0109cb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 --2- 172.21.15.53:0/3316740021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e0102a60 0x7f20e0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 --2- 172.21.15.53:0/3316740021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e0101870 0x7f20e0101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 -- 172.21.15.53:0/3316740021 >> 172.21.15.53:0/3316740021 conn(0x7f20e00fcde0 msgr2=0x7f20e00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:12.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 -- 172.21.15.53:0/3316740021 shutdown_connections 2024-04-28T12:48:12.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.251+0000 7f20e54a0640 1 -- 172.21.15.53:0/3316740021 wait complete. 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 Processor -- start 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 -- start start 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e0101870 0x7f20e006a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0102a60 0x7f20e019d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e0103420 0x7f20e019fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d380 con 0x7f20e0102a60 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d200 con 0x7f20e0101870 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.252+0000 7f20e54a0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d500 con 0x7f20e0103420 2024-04-28T12:48:12.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0102a60 0x7f20e019d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0102a60 0x7f20e019d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44884/0 (socket says 172.21.15.53:44884) 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 -- 172.21.15.53:0/153504664 learned_addr learned my addr 172.21.15.53:0/153504664 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20deffd640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e0101870 0x7f20e006a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20df7fe640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e0103420 0x7f20e019fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 -- 172.21.15.53:0/153504664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e0103420 msgr2=0x7f20e019fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e0103420 0x7f20e019fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 -- 172.21.15.53:0/153504664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e0101870 msgr2=0x7f20e006a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e0101870 0x7f20e006a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 -- 172.21.15.53:0/153504664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e01a0700 con 0x7f20e0102a60 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20df7fe640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e0103420 0x7f20e019fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20de7fc640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0102a60 0x7f20e019d800 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f20d400d950 tx=0x7f20d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20c7fff640 1 -- 172.21.15.53:0/153504664 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4017070 con 0x7f20e0102a60 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20c7fff640 1 -- 172.21.15.53:0/153504664 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20d4009c30 con 0x7f20e0102a60 2024-04-28T12:48:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e01a09f0 con 0x7f20e0102a60 2024-04-28T12:48:12.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20c7fff640 1 -- 172.21.15.53:0/153504664 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d400f620 con 0x7f20e0102a60 2024-04-28T12:48:12.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.253+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e01a0f40 con 0x7f20e0102a60 2024-04-28T12:48:12.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.254+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e0108b90 con 0x7f20e0102a60 2024-04-28T12:48:12.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.255+0000 7f20c7fff640 1 -- 172.21.15.53:0/153504664 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f20d4010460 con 0x7f20e0102a60 2024-04-28T12:48:12.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.256+0000 7f20c7fff640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f20c80a3bb0 0x7f20c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.256+0000 7f20deffd640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f20c80a3bb0 0x7f20c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.256+0000 7f20c7fff640 1 -- 172.21.15.53:0/153504664 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f20d40cfbb0 con 0x7f20e0102a60 2024-04-28T12:48:12.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.257+0000 7f20deffd640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f20c80a3bb0 0x7f20c80a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f20e01028c0 tx=0x7f20d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:12.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.261+0000 7f20c7fff640 1 -- 172.21.15.53:0/153504664 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f20d409ca10 con 0x7f20e0102a60 2024-04-28T12:48:12.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.491+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20e002dd30 con 0x7f20e0102a60 2024-04-28T12:48:12.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.492+0000 7f20c7fff640 1 -- 172.21.15.53:0/153504664 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f20d4009de0 con 0x7f20e0102a60 2024-04-28T12:48:12.487 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:12.488 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:48:12.488 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 4 2024-04-28T12:48:12.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.493+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f20c80a3bb0 msgr2=0x7f20c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.493+0000 7f20e54a0640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f20c80a3bb0 0x7f20c80a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f20e01028c0 tx=0x7f20d0005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.493+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0102a60 msgr2=0x7f20e019d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.493+0000 7f20e54a0640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0102a60 0x7f20e019d800 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f20d400d950 tx=0x7f20d400de10 comp rx=0 tx=0).stop 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 shutdown_connections 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f20c80a3bb0 0x7f20c80a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e0103420 0x7f20e019fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e0102a60 0x7f20e019d800 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 --2- 172.21.15.53:0/153504664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20e0101870 0x7f20e006a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 >> 172.21.15.53:0/153504664 conn(0x7f20e00fcde0 msgr2=0x7f20e0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 shutdown_connections 2024-04-28T12:48:12.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:12.494+0000 7f20e54a0640 1 -- 172.21.15.53:0/153504664 wait complete. 2024-04-28T12:48:12.500 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:12 smithi053 podman[69915]: 2024-04-28 12:48:12.248290347 +0000 UTC m=+0.375246928 container start 534b2f156e3af79d1adaa3c7c3f2e494046be293838e5b7d8b399abfeef69bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-28T12:48:12.500 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:12 smithi053 ceph-mds[69956]: -- 172.21.15.53:0/497020598 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560209ca3a40 con 0x560209c76c00 2024-04-28T12:48:12.500 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:12 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j[69930]: starting mds.j at 2024-04-28T12:48:12.500 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:12 smithi053 bash[69915]: 534b2f156e3af79d1adaa3c7c3f2e494046be293838e5b7d8b399abfeef69bb3 2024-04-28T12:48:12.500 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 12:48:12 smithi053 systemd[1]: Started Ceph mds.j for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:13.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: Deploying daemon mds.d on smithi053 2024-04-28T12:48:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.53:6834/1145263488,v1:172.21.15.53:6835/1145263488] up:boot 2024-04-28T12:48:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: fsmap 3 up:standby 2024-04-28T12:48:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T12:48:13.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:13 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/153504664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: Deploying daemon mds.d on smithi053 2024-04-28T12:48:13.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:13.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.53:6834/1145263488,v1:172.21.15.53:6835/1145263488] up:boot 2024-04-28T12:48:13.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: fsmap 3 up:standby 2024-04-28T12:48:13.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T12:48:13.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:13 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/153504664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:13.737 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.737 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.737 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:13.737 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:13.737 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:13.737 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:13.738 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: Deploying daemon mds.d on smithi053 2024-04-28T12:48:13.738 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:13.738 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.53:6834/1145263488,v1:172.21.15.53:6835/1145263488] up:boot 2024-04-28T12:48:13.738 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: fsmap 3 up:standby 2024-04-28T12:48:13.738 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T12:48:13.738 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:13 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/153504664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:14.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.691+0000 7f0452551640 1 -- 172.21.15.53:0/2878973922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c103420 msgr2=0x7f044c109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.691+0000 7f0452551640 1 --2- 172.21.15.53:0/2878973922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c103420 0x7f044c109cb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0444009ca0 tx=0x7f044402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 -- 172.21.15.53:0/2878973922 shutdown_connections 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 --2- 172.21.15.53:0/2878973922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c103420 0x7f044c109cb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 --2- 172.21.15.53:0/2878973922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c102a60 0x7f044c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 --2- 172.21.15.53:0/2878973922 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044c101870 0x7f044c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 -- 172.21.15.53:0/2878973922 >> 172.21.15.53:0/2878973922 conn(0x7f044c0fcde0 msgr2=0x7f044c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 -- 172.21.15.53:0/2878973922 shutdown_connections 2024-04-28T12:48:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 -- 172.21.15.53:0/2878973922 wait complete. 2024-04-28T12:48:14.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 Processor -- start 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.692+0000 7f0452551640 1 -- start start 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0452551640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c101870 0x7f044c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0452551640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044c102a60 0x7f044c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0452551640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c103420 0x7f044c1a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0452551640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f044c10d3a0 con 0x7f044c101870 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0452551640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f044c10d220 con 0x7f044c103420 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0452551640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f044c10d520 con 0x7f044c102a60 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c101870 0x7f044c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c101870 0x7f044c198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44928/0 (socket says 172.21.15.53:44928) 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 -- 172.21.15.53:0/2740720186 learned_addr learned my addr 172.21.15.53:0/2740720186 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044b7fe640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044c102a60 0x7f044c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0450aee640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c103420 0x7f044c1a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 -- 172.21.15.53:0/2740720186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044c102a60 msgr2=0x7f044c199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044c102a60 0x7f044c199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 -- 172.21.15.53:0/2740720186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c103420 msgr2=0x7f044c1a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c103420 0x7f044c1a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044bfff640 1 -- 172.21.15.53:0/2740720186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044c1a07a0 con 0x7f044c101870 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f044b7fe640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044c102a60 0x7f044c199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.693+0000 7f0450aee640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c103420 0x7f044c1a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.694+0000 7f044bfff640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c101870 0x7f044c198bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d950 tx=0x7f043c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.694+0000 7f04497fa640 1 -- 172.21.15.53:0/2740720186 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c017070 con 0x7f044c101870 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.694+0000 7f04497fa640 1 -- 172.21.15.53:0/2740720186 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f043c009c30 con 0x7f044c101870 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.694+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044c1a0a90 con 0x7f044c101870 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.694+0000 7f04497fa640 1 -- 172.21.15.53:0/2740720186 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c00f620 con 0x7f044c101870 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.694+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044c1a0f90 con 0x7f044c101870 2024-04-28T12:48:14.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.695+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044c069750 con 0x7f044c101870 2024-04-28T12:48:14.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.696+0000 7f04497fa640 1 -- 172.21.15.53:0/2740720186 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f043c01a070 con 0x7f044c101870 2024-04-28T12:48:14.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.697+0000 7f04497fa640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f04340a3bb0 0x7f04340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:14.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.697+0000 7f04497fa640 1 -- 172.21.15.53:0/2740720186 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f043c0cfa50 con 0x7f044c101870 2024-04-28T12:48:14.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.697+0000 7f044b7fe640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f04340a3bb0 0x7f04340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:14.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.697+0000 7f044b7fe640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f04340a3bb0 0x7f04340a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f044c19a090 tx=0x7f044000b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:14.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.702+0000 7f04497fa640 1 -- 172.21.15.53:0/2740720186 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f043c009de0 con 0x7f044c101870 2024-04-28T12:48:14.816 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:14 smithi053 systemd[1]: Starting Ceph mds.d for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:14.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.959+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f044c1a1450 con 0x7f044c101870 2024-04-28T12:48:14.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.960+0000 7f04497fa640 1 -- 172.21.15.53:0/2740720186 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f043c09c8d0 con 0x7f044c101870 2024-04-28T12:48:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:14.956 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:48:14.956 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 4 2024-04-28T12:48:14.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f04340a3bb0 msgr2=0x7f04340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:14.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f04340a3bb0 0x7f04340a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f044c19a090 tx=0x7f044000b5f0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c101870 msgr2=0x7f044c198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:14.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c101870 0x7f044c198bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d950 tx=0x7f043c00de10 comp rx=0 tx=0).stop 2024-04-28T12:48:14.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 shutdown_connections 2024-04-28T12:48:14.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f04340a3bb0 0x7f04340a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c103420 0x7f044c1a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f044c102a60 0x7f044c199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 --2- 172.21.15.53:0/2740720186 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f044c101870 0x7f044c198bf0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:14.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 >> 172.21.15.53:0/2740720186 conn(0x7f044c0fcde0 msgr2=0x7f044c0fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:14.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 shutdown_connections 2024-04-28T12:48:14.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:14.962+0000 7f0452551640 1 -- 172.21.15.53:0/2740720186 wait complete. 2024-04-28T12:48:15.116 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:14 smithi053 podman[70301]: 2024-04-28 12:48:14.758631955 +0000 UTC m=+0.018728546 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:15.116 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:14 smithi053 podman[70301]: 2024-04-28 12:48:14.989024686 +0000 UTC m=+0.249121251 container create 97fb01b2bc921be3971ff161c2a80f8fe7350102b550d12401a2bb854d4b4d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-28T12:48:15.497 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:15 smithi053 podman[70301]: 2024-04-28 12:48:15.241738751 +0000 UTC m=+0.501835317 container init 97fb01b2bc921be3971ff161c2a80f8fe7350102b550d12401a2bb854d4b4d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-28T12:48:15.497 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:15 smithi053 podman[70301]: 2024-04-28 12:48:15.244553061 +0000 UTC m=+0.504649627 container start 97fb01b2bc921be3971ff161c2a80f8fe7350102b550d12401a2bb854d4b4d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T12:48:15.497 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mds[70322]: -- 172.21.15.53:0/1138941307 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e189ffa40 con 0x556e189d3000 2024-04-28T12:48:15.498 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d[70318]: starting mds.d at 2024-04-28T12:48:15.498 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:15 smithi053 bash[70301]: 97fb01b2bc921be3971ff161c2a80f8fe7350102b550d12401a2bb854d4b4d2e 2024-04-28T12:48:15.498 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 12:48:15 smithi053 systemd[1]: Started Ceph mds.d for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2740720186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:15.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2740720186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:15.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:15.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:15.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2740720186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:15.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:15.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:15.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:15.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:16.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:16 smithi077 ceph-mon[32553]: Deploying daemon mds.e on smithi077 2024-04-28T12:48:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:16 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.53:6836/3609735401,v1:172.21.15.53:6837/3609735401] up:boot 2024-04-28T12:48:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:16 smithi077 ceph-mon[32553]: fsmap 4 up:standby 2024-04-28T12:48:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T12:48:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:16 smithi191 ceph-mon[33016]: Deploying daemon mds.e on smithi077 2024-04-28T12:48:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:16 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.53:6836/3609735401,v1:172.21.15.53:6837/3609735401] up:boot 2024-04-28T12:48:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:16 smithi191 ceph-mon[33016]: fsmap 4 up:standby 2024-04-28T12:48:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T12:48:16.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:16 smithi053 ceph-mon[29182]: Deploying daemon mds.e on smithi077 2024-04-28T12:48:16.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:16 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.53:6836/3609735401,v1:172.21.15.53:6837/3609735401] up:boot 2024-04-28T12:48:16.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:16 smithi053 ceph-mon[29182]: fsmap 4 up:standby 2024-04-28T12:48:16.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T12:48:16.970 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:17.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 -- 172.21.15.53:0/2605758969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd67006fa70 msgr2=0x7fd67006fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 --2- 172.21.15.53:0/2605758969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd67006fa70 0x7fd67006fed0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd660009900 tx=0x7fd66002a410 comp rx=0 tx=0).stop 2024-04-28T12:48:17.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 -- 172.21.15.53:0/2605758969 shutdown_connections 2024-04-28T12:48:17.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 --2- 172.21.15.53:0/2605758969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd670070410 0x7fd670070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 --2- 172.21.15.53:0/2605758969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd67006fa70 0x7fd67006fed0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 --2- 172.21.15.53:0/2605758969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd67006e830 0x7fd67006ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 -- 172.21.15.53:0/2605758969 >> 172.21.15.53:0/2605758969 conn(0x7fd67006d400 msgr2=0x7fd67006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:17.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 -- 172.21.15.53:0/2605758969 shutdown_connections 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.092+0000 7fd677862640 1 -- 172.21.15.53:0/2605758969 wait complete. 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 Processor -- start 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 -- start start 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd67006e830 0x7fd6701a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd67006fa70 0x7fd6701a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd670070410 0x7fd6701aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd67011d2a0 con 0x7fd670070410 2024-04-28T12:48:17.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd67011d120 con 0x7fd67006fa70 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd677862640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd67011d420 con 0x7fd67006e830 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd670070410 0x7fd6701aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd670070410 0x7fd6701aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35618/0 (socket says 172.21.15.53:35618) 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 -- 172.21.15.53:0/1766588676 learned_addr learned my addr 172.21.15.53:0/1766588676 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd674dfd640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd67006fa70 0x7fd6701a7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd6755fe640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd67006e830 0x7fd6701a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 -- 172.21.15.53:0/1766588676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd67006e830 msgr2=0x7fd6701a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd67006e830 0x7fd6701a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 -- 172.21.15.53:0/1766588676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd67006fa70 msgr2=0x7fd6701a7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd67006fa70 0x7fd6701a7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 -- 172.21.15.53:0/1766588676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6701af5f0 con 0x7fd670070410 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd674dfd640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd67006fa70 0x7fd6701a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd6755fe640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd67006e830 0x7fd6701a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.093+0000 7fd675dff640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd670070410 0x7fd6701aef80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd66800dbd0 tx=0x7fd66800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.094+0000 7fd6667fc640 1 -- 172.21.15.53:0/1766588676 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668003970 con 0x7fd670070410 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.094+0000 7fd6667fc640 1 -- 172.21.15.53:0/1766588676 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd668004020 con 0x7fd670070410 2024-04-28T12:48:17.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.094+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6701af8e0 con 0x7fd670070410 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.094+0000 7fd6667fc640 1 -- 172.21.15.53:0/1766588676 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680106c0 con 0x7fd670070410 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.094+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6701afde0 con 0x7fd670070410 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.095+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd670119d10 con 0x7fd670070410 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.096+0000 7fd6667fc640 1 -- 172.21.15.53:0/1766588676 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd6680041d0 con 0x7fd670070410 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.097+0000 7fd6667fc640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd65c0a3bb0 0x7fd65c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.097+0000 7fd6667fc640 1 -- 172.21.15.53:0/1766588676 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd6680d0180 con 0x7fd670070410 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.097+0000 7fd6755fe640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd65c0a3bb0 0x7fd65c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.097+0000 7fd6755fe640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd65c0a3bb0 0x7fd65c0a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005d00 tx=0x7fd66c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:17.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.101+0000 7fd6667fc640 1 -- 172.21.15.53:0/1766588676 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd668020030 con 0x7fd670070410 2024-04-28T12:48:17.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.325+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd67002dcd0 con 0x7fd670070410 2024-04-28T12:48:17.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.326+0000 7fd6667fc640 1 -- 172.21.15.53:0/1766588676 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7fd668017030 con 0x7fd670070410 2024-04-28T12:48:17.322 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:17.322 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:48:17.322 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 5 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd65c0a3bb0 msgr2=0x7fd65c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd65c0a3bb0 0x7fd65c0a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005d00 tx=0x7fd66c005c70 comp rx=0 tx=0).stop 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd670070410 msgr2=0x7fd6701aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd670070410 0x7fd6701aef80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd66800dbd0 tx=0x7fd66800b260 comp rx=0 tx=0).stop 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 shutdown_connections 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd65c0a3bb0 0x7fd65c0a6070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd670070410 0x7fd6701aef80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd67006fa70 0x7fd6701a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 --2- 172.21.15.53:0/1766588676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd67006e830 0x7fd6701a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 >> 172.21.15.53:0/1766588676 conn(0x7fd67006d400 msgr2=0x7fd67010b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:17.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 shutdown_connections 2024-04-28T12:48:17.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:17.328+0000 7fd677862640 1 -- 172.21.15.53:0/1766588676 wait complete. 2024-04-28T12:48:17.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:17 smithi053 ceph-mon[29182]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:17.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:17 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1766588676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:17 smithi191 ceph-mon[33016]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:17 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1766588676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:17.877 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:17 smithi077 ceph-mon[32553]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:17.877 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:17 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1766588676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:17.878 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:17 smithi077 systemd[1]: Starting Ceph mds.e for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:18.167 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:17 smithi077 podman[62401]: 2024-04-28 12:48:17.78539899 +0000 UTC m=+0.017619946 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:18.167 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:17 smithi077 podman[62401]: 2024-04-28 12:48:17.919720757 +0000 UTC m=+0.151941700 container create e1f58deb45467bc35e69cc5b381fa75420ae6e2e28c7915e06d5c5ae24d1a394 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-28T12:48:18.448 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:18 smithi077 podman[62401]: 2024-04-28 12:48:18.172633888 +0000 UTC m=+0.404854831 container init e1f58deb45467bc35e69cc5b381fa75420ae6e2e28c7915e06d5c5ae24d1a394 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-28T12:48:18.448 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:18 smithi077 podman[62401]: 2024-04-28 12:48:18.175276686 +0000 UTC m=+0.407497636 container start e1f58deb45467bc35e69cc5b381fa75420ae6e2e28c7915e06d5c5ae24d1a394 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:48:18.448 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:18 smithi077 ceph-mds[62420]: -- 172.21.15.77:0/2651686217 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555f1f529a40 con 0x555f1f4fcc00 2024-04-28T12:48:18.448 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e[62416]: starting mds.e at 2024-04-28T12:48:18.448 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:18 smithi077 bash[62401]: e1f58deb45467bc35e69cc5b381fa75420ae6e2e28c7915e06d5c5ae24d1a394 2024-04-28T12:48:18.448 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 12:48:18 smithi077 systemd[1]: Started Ceph mds.e for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:19.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:19.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 -- 172.21.15.53:0/2248379169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff478103b90 msgr2=0x7ff478104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:19.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 --2- 172.21.15.53:0/2248379169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff478103b90 0x7ff478104010 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009900 tx=0x7ff46c02a410 comp rx=0 tx=0).stop 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 -- 172.21.15.53:0/2248379169 shutdown_connections 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 --2- 172.21.15.53:0/2248379169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478104550 0x7ff47810ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 --2- 172.21.15.53:0/2248379169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff478103b90 0x7ff478104010 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 --2- 172.21.15.53:0/2248379169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4781029a0 0x7ff478102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 -- 172.21.15.53:0/2248379169 >> 172.21.15.53:0/2248379169 conn(0x7ff4780fdf30 msgr2=0x7ff478100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 -- 172.21.15.53:0/2248379169 shutdown_connections 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.455+0000 7ff47d771640 1 -- 172.21.15.53:0/2248379169 wait complete. 2024-04-28T12:48:19.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 Processor -- start 2024-04-28T12:48:19.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 -- start start 2024-04-28T12:48:19.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4781029a0 0x7ff478198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:19.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff478103b90 0x7ff478199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:19.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478104550 0x7ff4781a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:19.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff47810e4b0 con 0x7ff4781029a0 2024-04-28T12:48:19.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff47810e330 con 0x7ff478104550 2024-04-28T12:48:19.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff47d771640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff47810e630 con 0x7ff478103b90 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff476ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4781029a0 0x7ff478198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff476ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4781029a0 0x7ff478198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35638/0 (socket says 172.21.15.53:35638) 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff476ffd640 1 -- 172.21.15.53:0/1986201262 learned_addr learned my addr 172.21.15.53:0/1986201262 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.456+0000 7ff4767fc640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff478103b90 0x7ff478199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff4777fe640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478104550 0x7ff4781a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff476ffd640 1 -- 172.21.15.53:0/1986201262 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff478103b90 msgr2=0x7ff478199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff476ffd640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff478103b90 0x7ff478199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff476ffd640 1 -- 172.21.15.53:0/1986201262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478104550 msgr2=0x7ff4781a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff476ffd640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478104550 0x7ff4781a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff476ffd640 1 -- 172.21.15.53:0/1986201262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4781a0800 con 0x7ff4781029a0 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff476ffd640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4781029a0 0x7ff478198be0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff46800d950 tx=0x7ff46800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff45ffff640 1 -- 172.21.15.53:0/1986201262 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468017020 con 0x7ff4781029a0 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff45ffff640 1 -- 172.21.15.53:0/1986201262 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff468009c30 con 0x7ff4781029a0 2024-04-28T12:48:19.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4781a0af0 con 0x7ff4781029a0 2024-04-28T12:48:19.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff45ffff640 1 -- 172.21.15.53:0/1986201262 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46800f6b0 con 0x7ff4781029a0 2024-04-28T12:48:19.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.457+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4781a1060 con 0x7ff4781029a0 2024-04-28T12:48:19.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.458+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff478109de0 con 0x7ff4781029a0 2024-04-28T12:48:19.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.459+0000 7ff45ffff640 1 -- 172.21.15.53:0/1986201262 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff46801a020 con 0x7ff4781029a0 2024-04-28T12:48:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.460+0000 7ff45ffff640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff4600a3c30 0x7ff4600a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.460+0000 7ff4767fc640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff4600a3c30 0x7ff4600a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.460+0000 7ff45ffff640 1 -- 172.21.15.53:0/1986201262 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff4680cfac0 con 0x7ff4781029a0 2024-04-28T12:48:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.461+0000 7ff4767fc640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff4600a3c30 0x7ff4600a60f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff46c000c00 tx=0x7ff46c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:19.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.465+0000 7ff45ffff640 1 -- 172.21.15.53:0/1986201262 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff468009de0 con 0x7ff4781029a0 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: Deploying daemon mds.f on smithi191 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6834/1670611904,v1:172.21.15.77:6835/1670611904] up:boot 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: fsmap 5 up:standby 2024-04-28T12:48:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: Deploying daemon mds.f on smithi191 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6834/1670611904,v1:172.21.15.77:6835/1670611904] up:boot 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: fsmap 5 up:standby 2024-04-28T12:48:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T12:48:19.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.689+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff47802dd80 con 0x7ff4781029a0 2024-04-28T12:48:19.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.690+0000 7ff45ffff640 1 -- 172.21.15.53:0/1986201262 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7ff46809c940 con 0x7ff4781029a0 2024-04-28T12:48:19.685 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:19.686 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-28T12:48:19.686 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 6 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff4600a3c30 msgr2=0x7ff4600a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff4600a3c30 0x7ff4600a60f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff46c000c00 tx=0x7ff46c0314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4781029a0 msgr2=0x7ff478198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4781029a0 0x7ff478198be0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff46800d950 tx=0x7ff46800de10 comp rx=0 tx=0).stop 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 shutdown_connections 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff4600a3c30 0x7ff4600a60f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478104550 0x7ff4781a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff478103b90 0x7ff478199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 --2- 172.21.15.53:0/1986201262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4781029a0 0x7ff478198be0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 >> 172.21.15.53:0/1986201262 conn(0x7ff4780fdf30 msgr2=0x7ff478106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:19.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 shutdown_connections 2024-04-28T12:48:19.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:19.692+0000 7ff47d771640 1 -- 172.21.15.53:0/1986201262 wait complete. 2024-04-28T12:48:19.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: Deploying daemon mds.f on smithi191 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6834/1670611904,v1:172.21.15.77:6835/1670611904] up:boot 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: fsmap 5 up:standby 2024-04-28T12:48:19.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T12:48:20.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:20 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1986201262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:20.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:20 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1986201262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:20.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:20 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1986201262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:20.770 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:20 smithi191 systemd[1]: Starting Ceph mds.f for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:21.031 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:20 smithi191 podman[63555]: 2024-04-28 12:48:20.678024094 +0000 UTC m=+0.018061788 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:21.031 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:20 smithi191 podman[63555]: 2024-04-28 12:48:20.807263136 +0000 UTC m=+0.147300855 container create 01f68a3a61a2c66e885d220ff1fca0a1f094b67b2d866ea3c2982e15d5347925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-28T12:48:21.031 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:21 smithi191 podman[63555]: 2024-04-28 12:48:21.035683963 +0000 UTC m=+0.375721639 container init 01f68a3a61a2c66e885d220ff1fca0a1f094b67b2d866ea3c2982e15d5347925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-28T12:48:21.287 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:21 smithi191 podman[63555]: 2024-04-28 12:48:21.03826221 +0000 UTC m=+0.378299886 container start 01f68a3a61a2c66e885d220ff1fca0a1f094b67b2d866ea3c2982e15d5347925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-28T12:48:21.287 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:21 smithi191 ceph-mds[63574]: -- 172.21.15.191:0/3222587153 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a5d52da40 con 0x557a5d501000 2024-04-28T12:48:21.287 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f[63570]: starting mds.f at 2024-04-28T12:48:21.287 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:21 smithi191 bash[63555]: 01f68a3a61a2c66e885d220ff1fca0a1f094b67b2d866ea3c2982e15d5347925 2024-04-28T12:48:21.287 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 12:48:21 smithi191 systemd[1]: Started Ceph mds.f for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:21.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.819+0000 7f5521085640 1 -- 172.21.15.53:0/1182001775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 msgr2=0x7f551c104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.819+0000 7f5521085640 1 --2- 172.21.15.53:0/1182001775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c104140 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5504009900 tx=0x7f550402a400 comp rx=0 tx=0).stop 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 -- 172.21.15.53:0/1182001775 shutdown_connections 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 --2- 172.21.15.53:0/1182001775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f551c104680 0x7f551c10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 --2- 172.21.15.53:0/1182001775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c104140 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 --2- 172.21.15.53:0/1182001775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c100c50 0x7f551c101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 -- 172.21.15.53:0/1182001775 >> 172.21.15.53:0/1182001775 conn(0x7f551c0fc5a0 msgr2=0x7f551c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 -- 172.21.15.53:0/1182001775 shutdown_connections 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 -- 172.21.15.53:0/1182001775 wait complete. 2024-04-28T12:48:21.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.820+0000 7f5521085640 1 Processor -- start 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f5521085640 1 -- start start 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f5521085640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f551c100c50 0x7f551c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f5521085640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f5521085640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c104680 0x7f551c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f5521085640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f551c10e4c0 con 0x7f551c103cc0 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f5521085640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f551c10e340 con 0x7f551c104680 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f5521085640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f551c10e640 con 0x7f551c100c50 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35650/0 (socket says 172.21.15.53:35650) 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 -- 172.21.15.53:0/3173815378 learned_addr learned my addr 172.21.15.53:0/3173815378 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:21.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551b59e640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c104680 0x7f551c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551ad9d640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f551c100c50 0x7f551c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 -- 172.21.15.53:0/3173815378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f551c100c50 msgr2=0x7f551c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f551c100c50 0x7f551c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 -- 172.21.15.53:0/3173815378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c104680 msgr2=0x7f551c1a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c104680 0x7f551c1a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 -- 172.21.15.53:0/3173815378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c1a0810 con 0x7f551c103cc0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551ad9d640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f551c100c50 0x7f551c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.821+0000 7f551a59c640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c199130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5504005940 tx=0x7f55040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.822+0000 7f5513fff640 1 -- 172.21.15.53:0/3173815378 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550403f030 con 0x7f551c103cc0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.822+0000 7f5513fff640 1 -- 172.21.15.53:0/3173815378 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5504003cb0 con 0x7f551c103cc0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.822+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c1a0aa0 con 0x7f551c103cc0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.822+0000 7f5513fff640 1 -- 172.21.15.53:0/3173815378 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504035070 con 0x7f551c103cc0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.822+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c1a0fe0 con 0x7f551c103cc0 2024-04-28T12:48:21.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.823+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551c109e60 con 0x7f551c103cc0 2024-04-28T12:48:21.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.824+0000 7f5513fff640 1 -- 172.21.15.53:0/3173815378 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5504042030 con 0x7f551c103cc0 2024-04-28T12:48:21.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.824+0000 7f5513fff640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f55080a3c30 0x7f55080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:21.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.825+0000 7f551ad9d640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f55080a3c30 0x7f55080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:21.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.825+0000 7f5513fff640 1 -- 172.21.15.53:0/3173815378 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f55040f0b10 con 0x7f551c103cc0 2024-04-28T12:48:21.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.825+0000 7f551ad9d640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f55080a3c30 0x7f55080a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f551c06acf0 tx=0x7f550c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:21.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:21.828+0000 7f5513fff640 1 -- 172.21.15.53:0/3173815378 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55040bd8f0 con 0x7f551c103cc0 2024-04-28T12:48:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.054+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f551c02dc90 con 0x7f551c103cc0 2024-04-28T12:48:22.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.054+0000 7f5513fff640 1 -- 172.21.15.53:0/3173815378 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f5504003e60 con 0x7f551c103cc0 2024-04-28T12:48:22.050 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:22.050 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-28T12:48:22.050 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-04-28T12:48:22.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f55080a3c30 msgr2=0x7f55080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:22.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f55080a3c30 0x7f55080a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f551c06acf0 tx=0x7f550c005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:22.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 msgr2=0x7f551c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:22.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c199130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5504005940 tx=0x7f55040042f0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 shutdown_connections 2024-04-28T12:48:22.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f55080a3c30 0x7f55080a60f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c104680 0x7f551c1a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f551c103cc0 0x7f551c199130 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 --2- 172.21.15.53:0/3173815378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f551c100c50 0x7f551c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 >> 172.21.15.53:0/3173815378 conn(0x7f551c0fc5a0 msgr2=0x7f551c107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:22.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 shutdown_connections 2024-04-28T12:48:22.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:22.056+0000 7f5521085640 1 -- 172.21.15.53:0/3173815378 wait complete. 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: Deploying daemon mds.c on smithi191 2024-04-28T12:48:22.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6834/2819920278,v1:172.21.15.191:6835/2819920278] up:boot 2024-04-28T12:48:22.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: fsmap 6 up:standby 2024-04-28T12:48:22.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T12:48:22.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:22 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3173815378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:22.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: Deploying daemon mds.c on smithi191 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6834/2819920278,v1:172.21.15.191:6835/2819920278] up:boot 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: fsmap 6 up:standby 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T12:48:22.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:22 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3173815378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: Deploying daemon mds.c on smithi191 2024-04-28T12:48:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6834/2819920278,v1:172.21.15.191:6835/2819920278] up:boot 2024-04-28T12:48:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: fsmap 6 up:standby 2024-04-28T12:48:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T12:48:22.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:22 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3173815378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:23.691 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:23 smithi191 systemd[1]: Starting Ceph mds.c for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:23.988 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:23 smithi191 podman[63868]: 2024-04-28 12:48:23.598444011 +0000 UTC m=+0.017809673 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:23.988 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:23 smithi191 podman[63868]: 2024-04-28 12:48:23.730548314 +0000 UTC m=+0.149914005 container create 3cb94f0816f63f21bc8564eb3786b9e22273848bd305265f9afefe8013ba601d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-28T12:48:23.988 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:23 smithi191 podman[63868]: 2024-04-28 12:48:23.993009313 +0000 UTC m=+0.412374962 container init 3cb94f0816f63f21bc8564eb3786b9e22273848bd305265f9afefe8013ba601d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-28T12:48:24.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:24.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 -- 172.21.15.53:0/880618477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38101870 msgr2=0x7fde38101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 --2- 172.21.15.53:0/880618477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38101870 0x7fde38101c90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fde34009900 tx=0x7fde3402a410 comp rx=0 tx=0).stop 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 -- 172.21.15.53:0/880618477 shutdown_connections 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 --2- 172.21.15.53:0/880618477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38103420 0x7fde38109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 --2- 172.21.15.53:0/880618477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde38102a60 0x7fde38102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 --2- 172.21.15.53:0/880618477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38101870 0x7fde38101c90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 -- 172.21.15.53:0/880618477 >> 172.21.15.53:0/880618477 conn(0x7fde380fcde0 msgr2=0x7fde380ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 -- 172.21.15.53:0/880618477 shutdown_connections 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.186+0000 7fde40272640 1 -- 172.21.15.53:0/880618477 wait complete. 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 Processor -- start 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 -- start start 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38101870 0x7fde38105800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:24.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde38102a60 0x7fde38103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38103420 0x7fde38104380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde3810d370 con 0x7fde38103420 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fde3810d1f0 con 0x7fde38102a60 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde40272640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fde3810d4f0 con 0x7fde38101870 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde3e00e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38101870 0x7fde38105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde3e80f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38103420 0x7fde38104380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde3e00e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38101870 0x7fde38105800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.53:59378/0 (socket says 172.21.15.53:59378) 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde3e00e640 1 -- 172.21.15.53:0/3148232399 learned_addr learned my addr 172.21.15.53:0/3148232399 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.187+0000 7fde3d80d640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde38102a60 0x7fde38103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde3e00e640 1 -- 172.21.15.53:0/3148232399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde38102a60 msgr2=0x7fde38103e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde3e00e640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde38102a60 0x7fde38103e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde3e00e640 1 -- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38103420 msgr2=0x7fde38104380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde3e00e640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38103420 0x7fde38104380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde3e00e640 1 -- 172.21.15.53:0/3148232399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde38104c10 con 0x7fde38101870 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde3e80f640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38103420 0x7fde38104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde3e00e640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38101870 0x7fde38105800 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde34006010 tx=0x7fde340042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde2f7fe640 1 -- 172.21.15.53:0/3148232399 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3403f030 con 0x7fde38101870 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde2f7fe640 1 -- 172.21.15.53:0/3148232399 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde34031910 con 0x7fde38101870 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde381a2ce0 con 0x7fde38101870 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.188+0000 7fde2f7fe640 1 -- 172.21.15.53:0/3148232399 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34035070 con 0x7fde38101870 2024-04-28T12:48:24.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.189+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde381a3180 con 0x7fde38101870 2024-04-28T12:48:24.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.190+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde381a3350 con 0x7fde38101870 2024-04-28T12:48:24.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.191+0000 7fde2f7fe640 1 -- 172.21.15.53:0/3148232399 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fde34042030 con 0x7fde38101870 2024-04-28T12:48:24.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.191+0000 7fde2f7fe640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fde240a3bb0 0x7fde240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:24.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.191+0000 7fde3d80d640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fde240a3bb0 0x7fde240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:24.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.191+0000 7fde2f7fe640 1 -- 172.21.15.53:0/3148232399 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fde340f0520 con 0x7fde38101870 2024-04-28T12:48:24.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.192+0000 7fde3d80d640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fde240a3bb0 0x7fde240a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fde38105570 tx=0x7fde28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:24.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.196+0000 7fde2f7fe640 1 -- 172.21.15.53:0/3148232399 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fde340bd380 con 0x7fde38101870 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:24 smithi191 ceph-mon[33016]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:24 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:24 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:23 smithi191 podman[63868]: 2024-04-28 12:48:23.995681204 +0000 UTC m=+0.415046863 container start 3cb94f0816f63f21bc8564eb3786b9e22273848bd305265f9afefe8013ba601d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:24 smithi191 ceph-mds[63887]: -- 172.21.15.191:0/2346823616 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557335629a40 con 0x5573355fc800 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c[63883]: starting mds.c at 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:24 smithi191 bash[63868]: 3cb94f0816f63f21bc8564eb3786b9e22273848bd305265f9afefe8013ba601d 2024-04-28T12:48:24.287 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 12:48:24 smithi191 systemd[1]: Started Ceph mds.c for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:24.394 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:24 smithi053 ceph-mon[29182]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:24 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:24.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:24 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:24.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.424+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde381a3520 con 0x7fde38101870 2024-04-28T12:48:24.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.425+0000 7fde2f7fe640 1 -- 172.21.15.53:0/3148232399 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7fde34003e60 con 0x7fde38101870 2024-04-28T12:48:24.421 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:24.421 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-28T12:48:24.421 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 8 2024-04-28T12:48:24.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fde240a3bb0 msgr2=0x7fde240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:24.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fde240a3bb0 0x7fde240a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fde38105570 tx=0x7fde28005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:24.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38101870 msgr2=0x7fde38105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:24.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38101870 0x7fde38105800 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde34006010 tx=0x7fde340042f0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 shutdown_connections 2024-04-28T12:48:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fde240a3bb0 0x7fde240a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde38103420 0x7fde38104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde38102a60 0x7fde38103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 --2- 172.21.15.53:0/3148232399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38101870 0x7fde38105800 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 >> 172.21.15.53:0/3148232399 conn(0x7fde380fcde0 msgr2=0x7fde38069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 shutdown_connections 2024-04-28T12:48:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:24.427+0000 7fde40272640 1 -- 172.21.15.53:0/3148232399 wait complete. 2024-04-28T12:48:24.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:24 smithi077 ceph-mon[32553]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:24.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:24 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:24.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:24 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:25.190 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] up:boot 2024-04-28T12:48:25.191 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: fsmap 7 up:standby 2024-04-28T12:48:25.191 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T12:48:25.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:25.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:25.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: Deploying daemon mds.h on smithi077 2024-04-28T12:48:25.448 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:25 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3148232399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] up:boot 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: fsmap 7 up:standby 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: Deploying daemon mds.h on smithi077 2024-04-28T12:48:25.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:25 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3148232399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] up:boot 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: fsmap 7 up:standby 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:25.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: Deploying daemon mds.h on smithi077 2024-04-28T12:48:25.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:25 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3148232399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:26.372 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:26 smithi077 ceph-mon[32553]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:26.435 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:26.462 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:26 smithi053 ceph-mon[29182]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:26.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:26 smithi191 ceph-mon[33016]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 -- 172.21.15.53:0/4206332985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13701029a0 msgr2=0x7f1370102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 --2- 172.21.15.53:0/4206332985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13701029a0 0x7f1370102dc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1364009900 tx=0x7f136402a410 comp rx=0 tx=0).stop 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 -- 172.21.15.53:0/4206332985 shutdown_connections 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 --2- 172.21.15.53:0/4206332985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370104550 0x7f137010ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 --2- 172.21.15.53:0/4206332985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1370103b90 0x7f1370104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 --2- 172.21.15.53:0/4206332985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13701029a0 0x7f1370102dc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 -- 172.21.15.53:0/4206332985 >> 172.21.15.53:0/4206332985 conn(0x7f13700fdf30 msgr2=0x7f1370100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 -- 172.21.15.53:0/4206332985 shutdown_connections 2024-04-28T12:48:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.555+0000 7f1377fe7640 1 -- 172.21.15.53:0/4206332985 wait complete. 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 Processor -- start 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 -- start start 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13701029a0 0x7f1370198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370103b90 0x7f1370199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1370104550 0x7f13701a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f137010e520 con 0x7f1370104550 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f137010e3a0 con 0x7f13701029a0 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1377fe7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f137010e6a0 con 0x7f1370103b90 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1370104550 0x7f13701a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1370104550 0x7f13701a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35692/0 (socket says 172.21.15.53:35692) 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 -- 172.21.15.53:0/523963642 learned_addr learned my addr 172.21.15.53:0/523963642 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1375582640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370103b90 0x7f1370199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1375d83640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13701029a0 0x7f1370198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 -- 172.21.15.53:0/523963642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370103b90 msgr2=0x7f1370199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370103b90 0x7f1370199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 -- 172.21.15.53:0/523963642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13701029a0 msgr2=0x7f1370198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13701029a0 0x7f1370198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1376584640 1 -- 172.21.15.53:0/523963642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13701a0810 con 0x7f1370104550 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.556+0000 7f1375582640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370103b90 0x7f1370199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.557+0000 7f1376584640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1370104550 0x7f13701a01a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f136800eaa0 tx=0x7f136800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.557+0000 7f1362ffd640 1 -- 172.21.15.53:0/523963642 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136800cdb0 con 0x7f1370104550 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.557+0000 7f1362ffd640 1 -- 172.21.15.53:0/523963642 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1368005020 con 0x7f1370104550 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.557+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13701a0b00 con 0x7f1370104550 2024-04-28T12:48:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.557+0000 7f1362ffd640 1 -- 172.21.15.53:0/523963642 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13680116d0 con 0x7f1370104550 2024-04-28T12:48:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.557+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13701a1050 con 0x7f1370104550 2024-04-28T12:48:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.558+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1370109ce0 con 0x7f1370104550 2024-04-28T12:48:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.559+0000 7f1362ffd640 1 -- 172.21.15.53:0/523963642 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f13680051d0 con 0x7f1370104550 2024-04-28T12:48:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.559+0000 7f1362ffd640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f135c0a3bb0 0x7f135c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.560+0000 7f1375d83640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f135c0a3bb0 0x7f135c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.560+0000 7f1362ffd640 1 -- 172.21.15.53:0/523963642 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f13680d0590 con 0x7f1370104550 2024-04-28T12:48:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.560+0000 7f1375d83640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f135c0a3bb0 0x7f135c0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1364006010 tx=0x7f13640314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:26.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.563+0000 7f1362ffd640 1 -- 172.21.15.53:0/523963642 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1368016030 con 0x7f1370104550 2024-04-28T12:48:26.697 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:26 smithi077 systemd[1]: Starting Ceph mds.h for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:26.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.791+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f137002dd20 con 0x7f1370104550 2024-04-28T12:48:26.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.792+0000 7f1362ffd640 1 -- 172.21.15.53:0/523963642 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f136809d410 con 0x7f1370104550 2024-04-28T12:48:26.787 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:26.788 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-28T12:48:26.788 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 8 2024-04-28T12:48:26.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f135c0a3bb0 msgr2=0x7f135c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:26.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f135c0a3bb0 0x7f135c0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1364006010 tx=0x7f13640314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1370104550 msgr2=0x7f13701a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:26.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1370104550 0x7f13701a01a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f136800eaa0 tx=0x7f136800ef60 comp rx=0 tx=0).stop 2024-04-28T12:48:26.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 shutdown_connections 2024-04-28T12:48:26.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f135c0a3bb0 0x7f135c0a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1370104550 0x7f13701a01a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370103b90 0x7f1370199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 --2- 172.21.15.53:0/523963642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13701029a0 0x7f1370198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:26.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 >> 172.21.15.53:0/523963642 conn(0x7f13700fdf30 msgr2=0x7f1370106eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:26.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 shutdown_connections 2024-04-28T12:48:26.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:26.794+0000 7f1377fe7640 1 -- 172.21.15.53:0/523963642 wait complete. 2024-04-28T12:48:27.083 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:26 smithi077 podman[62708]: 2024-04-28 12:48:26.678584338 +0000 UTC m=+0.017893634 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:27.083 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:26 smithi077 podman[62708]: 2024-04-28 12:48:26.793407619 +0000 UTC m=+0.132716892 container create b8b012781aba50f01a86cb0ae8c9e2d633a3ef67124a89fd55a428fe0aa32d88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:48:27.083 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:27 smithi077 podman[62708]: 2024-04-28 12:48:27.088285053 +0000 UTC m=+0.427594335 container init b8b012781aba50f01a86cb0ae8c9e2d633a3ef67124a89fd55a428fe0aa32d88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-28T12:48:27.447 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:27 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/523963642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:27.448 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:27 smithi077 podman[62708]: 2024-04-28 12:48:27.09103241 +0000 UTC m=+0.430341687 container start b8b012781aba50f01a86cb0ae8c9e2d633a3ef67124a89fd55a428fe0aa32d88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:48:27.448 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:27 smithi077 ceph-mds[62727]: -- 172.21.15.77:0/1021419986 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55644cf2fa40 con 0x55644cf03000 2024-04-28T12:48:27.448 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:27 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h[62723]: starting mds.h at 2024-04-28T12:48:27.448 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:27 smithi077 bash[62708]: b8b012781aba50f01a86cb0ae8c9e2d633a3ef67124a89fd55a428fe0aa32d88 2024-04-28T12:48:27.448 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 12:48:27 smithi077 systemd[1]: Started Ceph mds.h for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:27.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:27 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/523963642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:27.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:27 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/523963642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] up:boot 2024-04-28T12:48:28.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: fsmap 8 up:standby 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:28.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:28 smithi191 ceph-mon[33016]: Deploying daemon mds.i on smithi191 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] up:boot 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: fsmap 8 up:standby 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:28.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:28.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:28.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:28 smithi053 ceph-mon[29182]: Deploying daemon mds.i on smithi191 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] up:boot 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: fsmap 8 up:standby 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:28 smithi077 ceph-mon[32553]: Deploying daemon mds.i on smithi191 2024-04-28T12:48:28.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 -- 172.21.15.53:0/2032843537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c101890 msgr2=0x7fba0c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 --2- 172.21.15.53:0/2032843537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c101890 0x7fba0c101cb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009900 tx=0x7fb9f402a410 comp rx=0 tx=0).stop 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 -- 172.21.15.53:0/2032843537 shutdown_connections 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 --2- 172.21.15.53:0/2032843537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0c103440 0x7fba0c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 --2- 172.21.15.53:0/2032843537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0c102a80 0x7fba0c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 --2- 172.21.15.53:0/2032843537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c101890 0x7fba0c101cb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 -- 172.21.15.53:0/2032843537 >> 172.21.15.53:0/2032843537 conn(0x7fba0c0fce20 msgr2=0x7fba0c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 -- 172.21.15.53:0/2032843537 shutdown_connections 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.921+0000 7fba12a1d640 1 -- 172.21.15.53:0/2032843537 wait complete. 2024-04-28T12:48:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 Processor -- start 2024-04-28T12:48:28.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 -- start start 2024-04-28T12:48:28.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0c101890 0x7fba0c1057d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c102a80 0x7fba0c103e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0c103440 0x7fba0c1043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c10d3c0 con 0x7fba0c102a80 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c10d240 con 0x7fba0c101890 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.922+0000 7fba12a1d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c10d540 con 0x7fba0c103440 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c102a80 0x7fba0c103e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c102a80 0x7fba0c103e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39282/0 (socket says 172.21.15.53:39282) 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 -- 172.21.15.53:0/3370050693 learned_addr learned my addr 172.21.15.53:0/3370050693 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0bfff640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0c101890 0x7fba0c1057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba10fba640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0c103440 0x7fba0c1043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 -- 172.21.15.53:0/3370050693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0c103440 msgr2=0x7fba0c1043a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0c103440 0x7fba0c1043a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 -- 172.21.15.53:0/3370050693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0c101890 msgr2=0x7fba0c1057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:28.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0c101890 0x7fba0c1057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 -- 172.21.15.53:0/3370050693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c104c60 con 0x7fba0c102a80 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba0b7fe640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c102a80 0x7fba0c103e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d950 tx=0x7fb9fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba097fa640 1 -- 172.21.15.53:0/3370050693 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc017070 con 0x7fba0c102a80 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba097fa640 1 -- 172.21.15.53:0/3370050693 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9fc009c30 con 0x7fba0c102a80 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c1a2d60 con 0x7fba0c102a80 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba097fa640 1 -- 172.21.15.53:0/3370050693 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00f620 con 0x7fba0c102a80 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.923+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c1a3220 con 0x7fba0c102a80 2024-04-28T12:48:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.924+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba0c1a3450 con 0x7fba0c102a80 2024-04-28T12:48:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.925+0000 7fba097fa640 1 -- 172.21.15.53:0/3370050693 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb9fc005030 con 0x7fba0c102a80 2024-04-28T12:48:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.926+0000 7fba097fa640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb9f80a3bb0 0x7fb9f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.926+0000 7fba0bfff640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb9f80a3bb0 0x7fb9f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.926+0000 7fba097fa640 1 -- 172.21.15.53:0/3370050693 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb9fc0cfb50 con 0x7fba0c102a80 2024-04-28T12:48:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.927+0000 7fba0bfff640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb9f80a3bb0 0x7fb9f80a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009230 tx=0x7fb9f40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:28.930+0000 7fba097fa640 1 -- 172.21.15.53:0/3370050693 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb9fc009de0 con 0x7fba0c102a80 2024-04-28T12:48:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.154+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba0c1a3620 con 0x7fba0c102a80 2024-04-28T12:48:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.155+0000 7fba097fa640 1 -- 172.21.15.53:0/3370050693 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fb9fc09c2e0 con 0x7fba0c102a80 2024-04-28T12:48:29.151 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:29.151 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-28T12:48:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 9 2024-04-28T12:48:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb9f80a3bb0 msgr2=0x7fb9f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb9f80a3bb0 0x7fb9f80a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009230 tx=0x7fb9f40314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c102a80 msgr2=0x7fba0c103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c102a80 0x7fba0c103e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d950 tx=0x7fb9fc00de10 comp rx=0 tx=0).stop 2024-04-28T12:48:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 shutdown_connections 2024-04-28T12:48:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb9f80a3bb0 0x7fb9f80a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba0c103440 0x7fba0c1043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c102a80 0x7fba0c103e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 --2- 172.21.15.53:0/3370050693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0c101890 0x7fba0c1057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 >> 172.21.15.53:0/3370050693 conn(0x7fba0c0fce20 msgr2=0x7fba0c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:29.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 shutdown_connections 2024-04-28T12:48:29.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:29.157+0000 7fba12a1d640 1 -- 172.21.15.53:0/3370050693 wait complete. 2024-04-28T12:48:29.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:29 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3370050693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:29.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:29 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3370050693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:29.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:29 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3370050693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:29.787 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:29 smithi191 podman[64178]: 2024-04-28 12:48:29.73169349 +0000 UTC m=+0.146152919 container create f60ee5b15a98dc8ab258c226d77fa14551d85cb55d64602f3366c1435cc7405d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T12:48:30.218 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:29 smithi191 podman[64178]: 2024-04-28 12:48:29.993604163 +0000 UTC m=+0.408063538 container init f60ee5b15a98dc8ab258c226d77fa14551d85cb55d64602f3366c1435cc7405d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-28T12:48:30.219 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:29 smithi191 podman[64178]: 2024-04-28 12:48:29.996244029 +0000 UTC m=+0.410703397 container start f60ee5b15a98dc8ab258c226d77fa14551d85cb55d64602f3366c1435cc7405d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-28T12:48:30.219 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mds[64197]: -- 172.21.15.191:0/3595459016 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5632ed495a40 con 0x5632ed469000 2024-04-28T12:48:30.219 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: starting mds.i at 2024-04-28T12:48:30.219 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:30 smithi191 bash[64178]: f60ee5b15a98dc8ab258c226d77fa14551d85cb55d64602f3366c1435cc7405d 2024-04-28T12:48:30.219 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:30 smithi191 systemd[1]: Started Ceph mds.i for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mon[33016]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:30 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:30.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:30 smithi053 ceph-mon[29182]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:30.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:30.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:30.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:30 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:30.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:30 smithi077 ceph-mon[32553]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:30.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:30 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:31.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:31.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 -- 172.21.15.53:0/2003971973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb881029a0 msgr2=0x7ffb88102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 --2- 172.21.15.53:0/2003971973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb881029a0 0x7ffb88102dc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009900 tx=0x7ffb7802a410 comp rx=0 tx=0).stop 2024-04-28T12:48:31.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 -- 172.21.15.53:0/2003971973 shutdown_connections 2024-04-28T12:48:31.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 --2- 172.21.15.53:0/2003971973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb88104550 0x7ffb8810ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 --2- 172.21.15.53:0/2003971973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb88103b90 0x7ffb88104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 --2- 172.21.15.53:0/2003971973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb881029a0 0x7ffb88102dc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 -- 172.21.15.53:0/2003971973 >> 172.21.15.53:0/2003971973 conn(0x7ffb880fdf30 msgr2=0x7ffb88100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 -- 172.21.15.53:0/2003971973 shutdown_connections 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.280+0000 7ffb8da25640 1 -- 172.21.15.53:0/2003971973 wait complete. 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 Processor -- start 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 -- start start 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb881029a0 0x7ffb88198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb88103b90 0x7ffb88199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb88104550 0x7ffb881a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8810e4d0 con 0x7ffb88104550 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8810e350 con 0x7ffb881029a0 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb8da25640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8810e650 con 0x7ffb88103b90 2024-04-28T12:48:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb877fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb88104550 0x7ffb881a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb877fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb88104550 0x7ffb881a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39308/0 (socket says 172.21.15.53:39308) 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.281+0000 7ffb877fe640 1 -- 172.21.15.53:0/1755075644 learned_addr learned my addr 172.21.15.53:0/1755075644 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb867fc640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb88103b90 0x7ffb88199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb86ffd640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb881029a0 0x7ffb88198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb877fe640 1 -- 172.21.15.53:0/1755075644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb88103b90 msgr2=0x7ffb88199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb877fe640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb88103b90 0x7ffb88199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb877fe640 1 -- 172.21.15.53:0/1755075644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb881029a0 msgr2=0x7ffb88198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb877fe640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb881029a0 0x7ffb88198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb877fe640 1 -- 172.21.15.53:0/1755075644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb881a0830 con 0x7ffb88104550 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb877fe640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb88104550 0x7ffb881a0190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000eb10 tx=0x7ffb8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb8ca23640 1 -- 172.21.15.53:0/1755075644 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80004b20 con 0x7ffb88104550 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb8ca23640 1 -- 172.21.15.53:0/1755075644 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb80004cc0 con 0x7ffb88104550 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb881a0b20 con 0x7ffb88104550 2024-04-28T12:48:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb8ca23640 1 -- 172.21.15.53:0/1755075644 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80010670 con 0x7ffb88104550 2024-04-28T12:48:31.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.282+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb881a10c0 con 0x7ffb88104550 2024-04-28T12:48:31.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.283+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb88109db0 con 0x7ffb88104550 2024-04-28T12:48:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.284+0000 7ffb8ca23640 1 -- 172.21.15.53:0/1755075644 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffb80005020 con 0x7ffb88104550 2024-04-28T12:48:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.285+0000 7ffb8ca23640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffb700a3c30 0x7ffb700a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.285+0000 7ffb86ffd640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffb700a3c30 0x7ffb700a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.285+0000 7ffb8ca23640 1 -- 172.21.15.53:0/1755075644 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ffb800d0610 con 0x7ffb88104550 2024-04-28T12:48:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.285+0000 7ffb86ffd640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffb700a3c30 0x7ffb700a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009230 tx=0x7ffb780314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:31.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.289+0000 7ffb8ca23640 1 -- 172.21.15.53:0/1755075644 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffb8009d490 con 0x7ffb88104550 2024-04-28T12:48:31.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:31 smithi053 ceph-mon[29182]: Deploying daemon mds.g on smithi053 2024-04-28T12:48:31.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:31 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] up:boot 2024-04-28T12:48:31.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:31 smithi053 ceph-mon[29182]: fsmap 9 up:standby 2024-04-28T12:48:31.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:31 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T12:48:31.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.517+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb8802dd20 con 0x7ffb88104550 2024-04-28T12:48:31.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.518+0000 7ffb8ca23640 1 -- 172.21.15.53:0/1755075644 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7ffb8009cd90 con 0x7ffb88104550 2024-04-28T12:48:31.514 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:31.514 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T12:48:31.514 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 10 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffb700a3c30 msgr2=0x7ffb700a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffb700a3c30 0x7ffb700a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009230 tx=0x7ffb780314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb88104550 msgr2=0x7ffb881a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb88104550 0x7ffb881a0190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000eb10 tx=0x7ffb8000c2d0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 shutdown_connections 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffb700a3c30 0x7ffb700a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb88104550 0x7ffb881a0190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb88103b90 0x7ffb88199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 --2- 172.21.15.53:0/1755075644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb881029a0 0x7ffb88198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.520+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 >> 172.21.15.53:0/1755075644 conn(0x7ffb880fdf30 msgr2=0x7ffb88106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.521+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 shutdown_connections 2024-04-28T12:48:31.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:31.521+0000 7ffb8da25640 1 -- 172.21.15.53:0/1755075644 wait complete. 2024-04-28T12:48:31.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:31 smithi191 ceph-mon[33016]: Deploying daemon mds.g on smithi053 2024-04-28T12:48:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:31 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] up:boot 2024-04-28T12:48:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:31 smithi191 ceph-mon[33016]: fsmap 9 up:standby 2024-04-28T12:48:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:31 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T12:48:31.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:31 smithi077 ceph-mon[32553]: Deploying daemon mds.g on smithi053 2024-04-28T12:48:31.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:31 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] up:boot 2024-04-28T12:48:31.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:31 smithi077 ceph-mon[32553]: fsmap 9 up:standby 2024-04-28T12:48:31.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:31 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T12:48:32.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:32 smithi053 ceph-mon[29182]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:32.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:32 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1755075644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:32.395 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 systemd[1]: Starting Ceph mds.g for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:32 smithi191 ceph-mon[33016]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:32.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:32 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1755075644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:32.697 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:32 smithi077 ceph-mon[32553]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:32.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:32 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1755075644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:32.767 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 podman[70899]: 2024-04-28 12:48:32.41038539 +0000 UTC m=+0.019186383 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:32.767 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 podman[70899]: 2024-04-28 12:48:32.519531521 +0000 UTC m=+0.128332501 container create ce9d304d4420a152597eea303d5bf8d24fa9889fb19de0aabe21c1ea65168be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T12:48:33.145 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 podman[70899]: 2024-04-28 12:48:32.772176522 +0000 UTC m=+0.380977503 container init ce9d304d4420a152597eea303d5bf8d24fa9889fb19de0aabe21c1ea65168be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-28T12:48:33.145 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 podman[70899]: 2024-04-28 12:48:32.775065023 +0000 UTC m=+0.383866008 container start ce9d304d4420a152597eea303d5bf8d24fa9889fb19de0aabe21c1ea65168be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-28T12:48:33.145 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 ceph-mds[70919]: -- 172.21.15.53:0/3127205517 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565144de1a40 con 0x565144db4800 2024-04-28T12:48:33.145 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g[70915]: starting mds.g at 2024-04-28T12:48:33.145 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 bash[70899]: ce9d304d4420a152597eea303d5bf8d24fa9889fb19de0aabe21c1ea65168be7 2024-04-28T12:48:33.145 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 12:48:32 smithi053 systemd[1]: Started Ceph mds.g for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:33.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.638+0000 7f864bd7f640 1 -- 172.21.15.53:0/2822920274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 msgr2=0x7f8644105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.638+0000 7f864bd7f640 1 --2- 172.21.15.53:0/2822920274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644105800 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8640009900 tx=0x7f864002a410 comp rx=0 tx=0).stop 2024-04-28T12:48:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.638+0000 7f864bd7f640 1 -- 172.21.15.53:0/2822920274 shutdown_connections 2024-04-28T12:48:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.638+0000 7f864bd7f640 1 --2- 172.21.15.53:0/2822920274 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8644108670 0x7f864410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.638+0000 7f864bd7f640 1 --2- 172.21.15.53:0/2822920274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8644105d40 0x7f8644108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.638+0000 7f864bd7f640 1 --2- 172.21.15.53:0/2822920274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644105800 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.638+0000 7f864bd7f640 1 -- 172.21.15.53:0/2822920274 >> 172.21.15.53:0/2822920274 conn(0x7f86440fce20 msgr2=0x7f86440ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 -- 172.21.15.53:0/2822920274 shutdown_connections 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 -- 172.21.15.53:0/2822920274 wait complete. 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 Processor -- start 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 -- start start 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8644105d40 0x7f86441990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8644108670 0x7f86441a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f864410e4c0 con 0x7f8644103410 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f864410e340 con 0x7f8644105d40 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f864bd7f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f864410e640 con 0x7f8644108670 2024-04-28T12:48:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f8649b1b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f8649b1b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39346/0 (socket says 172.21.15.53:39346) 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.639+0000 7f8649b1b640 1 -- 172.21.15.53:0/1932403526 learned_addr learned my addr 172.21.15.53:0/1932403526 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f864931a640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8644105d40 0x7f86441990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f864a31c640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8644108670 0x7f86441a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f8649b1b640 1 -- 172.21.15.53:0/1932403526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8644108670 msgr2=0x7f86441a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f8649b1b640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8644108670 0x7f86441a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f8649b1b640 1 -- 172.21.15.53:0/1932403526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8644105d40 msgr2=0x7f86441990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f8649b1b640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8644105d40 0x7f86441990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f8649b1b640 1 -- 172.21.15.53:0/1932403526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86441a07b0 con 0x7f8644103410 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f864a31c640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8644108670 0x7f86441a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f864931a640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8644105d40 0x7f86441990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f8649b1b640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644198b90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8640005940 tx=0x7f8640004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f863affd640 1 -- 172.21.15.53:0/1932403526 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864003f030 con 0x7f8644103410 2024-04-28T12:48:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f863affd640 1 -- 172.21.15.53:0/1932403526 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8640003cb0 con 0x7f8644103410 2024-04-28T12:48:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86441a0a40 con 0x7f8644103410 2024-04-28T12:48:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f863affd640 1 -- 172.21.15.53:0/1932403526 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640035070 con 0x7f8644103410 2024-04-28T12:48:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.640+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86441a0f80 con 0x7f8644103410 2024-04-28T12:48:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.641+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86440fe910 con 0x7f8644103410 2024-04-28T12:48:33.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.643+0000 7f863affd640 1 -- 172.21.15.53:0/1932403526 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8640042030 con 0x7f8644103410 2024-04-28T12:48:33.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.644+0000 7f863affd640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86300a3c30 0x7f86300a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.644+0000 7f864931a640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86300a3c30 0x7f86300a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.644+0000 7f863affd640 1 -- 172.21.15.53:0/1932403526 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f86400f0ca0 con 0x7f8644103410 2024-04-28T12:48:33.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.644+0000 7f864931a640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86300a3c30 0x7f86300a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f86340058c0 tx=0x7f8634005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:33.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.648+0000 7f863affd640 1 -- 172.21.15.53:0/1932403526 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f86400bda80 con 0x7f8644103410 2024-04-28T12:48:33.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.880+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f864402dad0 con 0x7f8644103410 2024-04-28T12:48:33.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.880+0000 7f863affd640 1 -- 172.21.15.53:0/1932403526 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f86400d1e20 con 0x7f8644103410 2024-04-28T12:48:33.877 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:33.877 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T12:48:33.877 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 11 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86300a3c30 msgr2=0x7f86300a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86300a3c30 0x7f86300a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f86340058c0 tx=0x7f8634005850 comp rx=0 tx=0).stop 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 msgr2=0x7f8644198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644198b90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8640005940 tx=0x7f8640004430 comp rx=0 tx=0).stop 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 shutdown_connections 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86300a3c30 0x7f86300a60f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8644108670 0x7f86441a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8644105d40 0x7f86441990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 --2- 172.21.15.53:0/1932403526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8644103410 0x7f8644198b90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 >> 172.21.15.53:0/1932403526 conn(0x7f86440fce20 msgr2=0x7f8644103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:33.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 shutdown_connections 2024-04-28T12:48:33.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:33.883+0000 7f864bd7f640 1 -- 172.21.15.53:0/1932403526 wait complete. 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: Deploying daemon mds.a on smithi053 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.53:6838/2476446586,v1:172.21.15.53:6839/2476446586] up:boot 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: fsmap 10 up:standby 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T12:48:34.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:33 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1932403526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:34.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:34.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: Deploying daemon mds.a on smithi053 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.53:6838/2476446586,v1:172.21.15.53:6839/2476446586] up:boot 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: fsmap 10 up:standby 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T12:48:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:33 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1932403526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: Deploying daemon mds.a on smithi053 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.53:6838/2476446586,v1:172.21.15.53:6839/2476446586] up:boot 2024-04-28T12:48:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: fsmap 10 up:standby 2024-04-28T12:48:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T12:48:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:33 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1932403526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:35.145 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 systemd[1]: Starting Ceph mds.a for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T12:48:35.531 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 podman[71257]: 2024-04-28 12:48:35.178936041 +0000 UTC m=+0.019102027 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:35.532 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 podman[71257]: 2024-04-28 12:48:35.291547881 +0000 UTC m=+0.131713866 container create 9bc0ae78e38689613db5523a4295e8120a8e2a2a334e70e8e7151573fe93e674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:48:35.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:35.894 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 podman[71257]: 2024-04-28 12:48:35.536291956 +0000 UTC m=+0.376457944 container init 9bc0ae78e38689613db5523a4295e8120a8e2a2a334e70e8e7151573fe93e674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:48:35.895 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 podman[71257]: 2024-04-28 12:48:35.539451207 +0000 UTC m=+0.379617195 container start 9bc0ae78e38689613db5523a4295e8120a8e2a2a334e70e8e7151573fe93e674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-28T12:48:35.895 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mds[71277]: -- 172.21.15.53:0/1614729829 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0ad735a40 con 0x55f0ad709000 2024-04-28T12:48:35.895 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a[71272]: starting mds.a at 2024-04-28T12:48:35.895 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 bash[71257]: 9bc0ae78e38689613db5523a4295e8120a8e2a2a334e70e8e7151573fe93e674 2024-04-28T12:48:35.895 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 12:48:35 smithi053 systemd[1]: Started Ceph mds.a for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 -- 172.21.15.53:0/3717880701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc105dc0 msgr2=0x7f96bc1081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 --2- 172.21.15.53:0/3717880701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc105dc0 0x7f96bc1081b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009900 tx=0x7f96b002a410 comp rx=0 tx=0).stop 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 -- 172.21.15.53:0/3717880701 shutdown_connections 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 --2- 172.21.15.53:0/3717880701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96bc1086f0 0x7f96bc10aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 --2- 172.21.15.53:0/3717880701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc105dc0 0x7f96bc1081b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 --2- 172.21.15.53:0/3717880701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96bc103490 0x7f96bc105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 -- 172.21.15.53:0/3717880701 >> 172.21.15.53:0/3717880701 conn(0x7f96bc0fcde0 msgr2=0x7f96bc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 -- 172.21.15.53:0/3717880701 shutdown_connections 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.003+0000 7f96c2380640 1 -- 172.21.15.53:0/3717880701 wait complete. 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 Processor -- start 2024-04-28T12:48:36.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 -- start start 2024-04-28T12:48:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc103490 0x7f96bc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96bc105dc0 0x7f96bc199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96bc1086f0 0x7f96bc1a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc10e470 con 0x7f96bc103490 2024-04-28T12:48:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc10e2f0 con 0x7f96bc105dc0 2024-04-28T12:48:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96c2380640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc10e5f0 con 0x7f96bc1086f0 2024-04-28T12:48:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc103490 0x7f96bc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96bbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc103490 0x7f96bc198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39378/0 (socket says 172.21.15.53:39378) 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96bbfff640 1 -- 172.21.15.53:0/4164786212 learned_addr learned my addr 172.21.15.53:0/4164786212 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.004+0000 7f96bb7fe640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96bc105dc0 0x7f96bc199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96c091d640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96bc1086f0 0x7f96bc1a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96bbfff640 1 -- 172.21.15.53:0/4164786212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96bc1086f0 msgr2=0x7f96bc1a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96bbfff640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96bc1086f0 0x7f96bc1a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96bbfff640 1 -- 172.21.15.53:0/4164786212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96bc105dc0 msgr2=0x7f96bc199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96bbfff640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96bc105dc0 0x7f96bc199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96bbfff640 1 -- 172.21.15.53:0/4164786212 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc1a07e0 con 0x7f96bc103490 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96c091d640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96bc1086f0 0x7f96bc1a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96bbfff640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc103490 0x7f96bc198c10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00e9f0 tx=0x7f96ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96b97fa640 1 -- 172.21.15.53:0/4164786212 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00cd80 con 0x7f96bc103490 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96b97fa640 1 -- 172.21.15.53:0/4164786212 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96ac0108e0 con 0x7f96bc103490 2024-04-28T12:48:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96b97fa640 1 -- 172.21.15.53:0/4164786212 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac018980 con 0x7f96bc103490 2024-04-28T12:48:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96bc1a0ad0 con 0x7f96bc103490 2024-04-28T12:48:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.005+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96bc101510 con 0x7f96bc103490 2024-04-28T12:48:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.006+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96bc0fe8d0 con 0x7f96bc103490 2024-04-28T12:48:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.007+0000 7f96b97fa640 1 -- 172.21.15.53:0/4164786212 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f96ac010430 con 0x7f96bc103490 2024-04-28T12:48:36.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.008+0000 7f96b97fa640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f96a40a3c30 0x7f96a40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.008+0000 7f96bb7fe640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f96a40a3c30 0x7f96a40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.008+0000 7f96b97fa640 1 -- 172.21.15.53:0/4164786212 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f96ac0d07f0 con 0x7f96bc103490 2024-04-28T12:48:36.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.008+0000 7f96bb7fe640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f96a40a3c30 0x7f96a40a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009230 tx=0x7f96b00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:36.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.012+0000 7f96b97fa640 1 -- 172.21.15.53:0/4164786212 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f96ac09d5d0 con 0x7f96bc103490 2024-04-28T12:48:36.197 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:35 smithi077 ceph-mon[32553]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:36.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.236+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96bc02dad0 con 0x7f96bc103490 2024-04-28T12:48:36.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.237+0000 7f96b97fa640 1 -- 172.21.15.53:0/4164786212 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7f96ac09ced0 con 0x7f96bc103490 2024-04-28T12:48:36.233 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:36.233 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T12:48:36.233 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 12 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f96a40a3c30 msgr2=0x7f96a40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f96a40a3c30 0x7f96a40a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009230 tx=0x7f96b00314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc103490 msgr2=0x7f96bc198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc103490 0x7f96bc198c10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00e9f0 tx=0x7f96ac00eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 shutdown_connections 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f96a40a3c30 0x7f96a40a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96bc1086f0 0x7f96bc1a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96bc105dc0 0x7f96bc199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 --2- 172.21.15.53:0/4164786212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96bc103490 0x7f96bc198c10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 >> 172.21.15.53:0/4164786212 conn(0x7f96bc0fcde0 msgr2=0x7f96bc103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:36.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 shutdown_connections 2024-04-28T12:48:36.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:36.239+0000 7f96c2380640 1 -- 172.21.15.53:0/4164786212 wait complete. 2024-04-28T12:48:36.245 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mon[29182]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:36.245 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.245 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.245 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.245 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:36.245 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:36.245 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:36.286 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:35 smithi191 ceph-mon[33016]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:37.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:36 smithi077 ceph-mon[32553]: Deploying daemon mds.b on smithi077 2024-04-28T12:48:37.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:36 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.53:6840/1978797303,v1:172.21.15.53:6841/1978797303] up:boot 2024-04-28T12:48:37.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:36 smithi077 ceph-mon[32553]: fsmap 11 up:standby 2024-04-28T12:48:37.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:36 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T12:48:37.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:36 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4164786212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:36 smithi191 ceph-mon[33016]: Deploying daemon mds.b on smithi077 2024-04-28T12:48:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:36 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.53:6840/1978797303,v1:172.21.15.53:6841/1978797303] up:boot 2024-04-28T12:48:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:36 smithi191 ceph-mon[33016]: fsmap 11 up:standby 2024-04-28T12:48:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:36 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T12:48:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:36 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4164786212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:36 smithi053 ceph-mon[29182]: Deploying daemon mds.b on smithi077 2024-04-28T12:48:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:36 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.53:6840/1978797303,v1:172.21.15.53:6841/1978797303] up:boot 2024-04-28T12:48:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:36 smithi053 ceph-mon[29182]: fsmap 11 up:standby 2024-04-28T12:48:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:36 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T12:48:37.395 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:36 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4164786212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:38.106 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:37 smithi077 ceph-mon[32553]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:38.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:38.274 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:37 smithi053 ceph-mon[29182]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:38.286 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:37 smithi191 ceph-mon[33016]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 -- 172.21.15.53:0/2948400580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 msgr2=0x7fe384105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 --2- 172.21.15.53:0/2948400580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384105800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe374009900 tx=0x7fe37402a410 comp rx=0 tx=0).stop 2024-04-28T12:48:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 -- 172.21.15.53:0/2948400580 shutdown_connections 2024-04-28T12:48:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 --2- 172.21.15.53:0/2948400580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe384108670 0x7fe38410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 --2- 172.21.15.53:0/2948400580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe384105d40 0x7fe384108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 --2- 172.21.15.53:0/2948400580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384105800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 -- 172.21.15.53:0/2948400580 >> 172.21.15.53:0/2948400580 conn(0x7fe3840fce20 msgr2=0x7fe3840ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 -- 172.21.15.53:0/2948400580 shutdown_connections 2024-04-28T12:48:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.367+0000 7fe389112640 1 -- 172.21.15.53:0/2948400580 wait complete. 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 Processor -- start 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 -- start start 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe384105d40 0x7fe3841991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe384108670 0x7fe3841a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe38410e4f0 con 0x7fe384103410 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe38410e370 con 0x7fe384105d40 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe389112640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe38410e670 con 0x7fe384108670 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57268/0 (socket says 172.21.15.53:57268) 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 -- 172.21.15.53:0/1135803098 learned_addr learned my addr 172.21.15.53:0/1135803098 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe38359e640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe384108670 0x7fe3841a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe38259c640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe384105d40 0x7fe3841991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 -- 172.21.15.53:0/1135803098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe384108670 msgr2=0x7fe3841a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe384108670 0x7fe3841a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 -- 172.21.15.53:0/1135803098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe384105d40 msgr2=0x7fe3841991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe384105d40 0x7fe3841991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.368+0000 7fe382d9d640 1 -- 172.21.15.53:0/1135803098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3841a0890 con 0x7fe384103410 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe38359e640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe384108670 0x7fe3841a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe38259c640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe384105d40 0x7fe3841991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe382d9d640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384198c80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe374005940 tx=0x7fe374031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:38.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe36bfff640 1 -- 172.21.15.53:0/1135803098 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37403f030 con 0x7fe384103410 2024-04-28T12:48:38.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3841a0b20 con 0x7fe384103410 2024-04-28T12:48:38.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe36bfff640 1 -- 172.21.15.53:0/1135803098 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe374003aa0 con 0x7fe384103410 2024-04-28T12:48:38.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe36bfff640 1 -- 172.21.15.53:0/1135803098 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374035070 con 0x7fe384103410 2024-04-28T12:48:38.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.369+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3841a1060 con 0x7fe384103410 2024-04-28T12:48:38.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.370+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3840fe910 con 0x7fe384103410 2024-04-28T12:48:38.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.371+0000 7fe36bfff640 1 -- 172.21.15.53:0/1135803098 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe374042030 con 0x7fe384103410 2024-04-28T12:48:38.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.372+0000 7fe36bfff640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe36c0a3bb0 0x7fe36c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.372+0000 7fe38259c640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe36c0a3bb0 0x7fe36c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.372+0000 7fe36bfff640 1 -- 172.21.15.53:0/1135803098 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe3740f0780 con 0x7fe384103410 2024-04-28T12:48:38.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.372+0000 7fe38259c640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe36c0a3bb0 0x7fe36c0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe38406baf0 tx=0x7fe378005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:38.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.376+0000 7fe36bfff640 1 -- 172.21.15.53:0/1135803098 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe3740bd560 con 0x7fe384103410 2024-04-28T12:48:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.607+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe384063530 con 0x7fe384103410 2024-04-28T12:48:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.608+0000 7fe36bfff640 1 -- 172.21.15.53:0/1135803098 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7fe3740bce60 con 0x7fe384103410 2024-04-28T12:48:38.604 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:38.604 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T12:48:38.604 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 12 2024-04-28T12:48:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe36c0a3bb0 msgr2=0x7fe36c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe36c0a3bb0 0x7fe36c0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe38406baf0 tx=0x7fe378005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 msgr2=0x7fe384198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384198c80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe374005940 tx=0x7fe374031a50 comp rx=0 tx=0).stop 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 shutdown_connections 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe36c0a3bb0 0x7fe36c0a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe384108670 0x7fe3841a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe384105d40 0x7fe3841991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 --2- 172.21.15.53:0/1135803098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe384103410 0x7fe384198c80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 >> 172.21.15.53:0/1135803098 conn(0x7fe3840fce20 msgr2=0x7fe384103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 shutdown_connections 2024-04-28T12:48:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:38.610+0000 7fe389112640 1 -- 172.21.15.53:0/1135803098 wait complete. 2024-04-28T12:48:38.697 INFO:journalctl@ceph.mds.b.smithi077.stdout:Apr 28 12:48:38 smithi077 systemd[1]: Started Ceph mds.b for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1135803098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:39.818 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6838/2957868311,v1:172.21.15.77:6839/2957868311] up:boot 2024-04-28T12:48:39.819 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: fsmap 12 up:standby 2024-04-28T12:48:39.819 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T12:48:39.819 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:39 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:39.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1135803098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6838/2957868311,v1:172.21.15.77:6839/2957868311] up:boot 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: fsmap 12 up:standby 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T12:48:39.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:39 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1135803098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6838/2957868311,v1:172.21.15.77:6839/2957868311] up:boot 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: fsmap 12 up:standby 2024-04-28T12:48:39.836 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T12:48:39.837 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:39 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:40.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:40.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 -- 172.21.15.53:0/1638465017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68101870 msgr2=0x7fca68101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1638465017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68101870 0x7fca68101c90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009860 tx=0x7fca5c02a340 comp rx=0 tx=0).stop 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 -- 172.21.15.53:0/1638465017 shutdown_connections 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1638465017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca68103420 0x7fca68109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1638465017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca68102a60 0x7fca68102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1638465017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68101870 0x7fca68101c90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 -- 172.21.15.53:0/1638465017 >> 172.21.15.53:0/1638465017 conn(0x7fca680fcde0 msgr2=0x7fca680ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 -- 172.21.15.53:0/1638465017 shutdown_connections 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.728+0000 7fca6ee37640 1 -- 172.21.15.53:0/1638465017 wait complete. 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 Processor -- start 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 -- start start 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca68101870 0x7fca6806a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68102a60 0x7fca6819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca68103420 0x7fca6819ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fca6810d3a0 con 0x7fca68102a60 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fca6810d220 con 0x7fca68103420 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6ee37640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fca6810d520 con 0x7fca68101870 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca67fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68102a60 0x7fca6819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca67fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68102a60 0x7fca6819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57288/0 (socket says 172.21.15.53:57288) 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca67fff640 1 -- 172.21.15.53:0/1163815164 learned_addr learned my addr 172.21.15.53:0/1163815164 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.729+0000 7fca6cbd3640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca68101870 0x7fca6806a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca6d3d4640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca68103420 0x7fca6819ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca67fff640 1 -- 172.21.15.53:0/1163815164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca68101870 msgr2=0x7fca6806a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca67fff640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca68101870 0x7fca6806a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca67fff640 1 -- 172.21.15.53:0/1163815164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca68103420 msgr2=0x7fca6819ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca67fff640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca68103420 0x7fca6819ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca67fff640 1 -- 172.21.15.53:0/1163815164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca681a0800 con 0x7fca68102a60 2024-04-28T12:48:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca6d3d4640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca68103420 0x7fca6819ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:40.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca67fff640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68102a60 0x7fca6819d900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fca5400ea60 tx=0x7fca5400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:40.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca65ffb640 1 -- 172.21.15.53:0/1163815164 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5400cde0 con 0x7fca68102a60 2024-04-28T12:48:40.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca65ffb640 1 -- 172.21.15.53:0/1163815164 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca540035f0 con 0x7fca68102a60 2024-04-28T12:48:40.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca681a0af0 con 0x7fca68102a60 2024-04-28T12:48:40.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca65ffb640 1 -- 172.21.15.53:0/1163815164 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca54010690 con 0x7fca68102a60 2024-04-28T12:48:40.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.730+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca681a1040 con 0x7fca68102a60 2024-04-28T12:48:40.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.731+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca68108c10 con 0x7fca68102a60 2024-04-28T12:48:40.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.732+0000 7fca65ffb640 1 -- 172.21.15.53:0/1163815164 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fca540037a0 con 0x7fca68102a60 2024-04-28T12:48:40.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.733+0000 7fca65ffb640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fca580a3bb0 0x7fca580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.733+0000 7fca6cbd3640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fca580a3bb0 0x7fca580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.733+0000 7fca65ffb640 1 -- 172.21.15.53:0/1163815164 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fca540d0340 con 0x7fca68102a60 2024-04-28T12:48:40.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.733+0000 7fca6cbd3640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fca580a3bb0 0x7fca580a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009830 tx=0x7fca5c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:40.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.738+0000 7fca65ffb640 1 -- 172.21.15.53:0/1163815164 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fca5409d120 con 0x7fca68102a60 2024-04-28T12:48:40.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.973+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca68063530 con 0x7fca68102a60 2024-04-28T12:48:40.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.973+0000 7fca65ffb640 1 -- 172.21.15.53:0/1163815164 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7fca5409ca20 con 0x7fca68102a60 2024-04-28T12:48:40.969 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:40.969 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T12:48:40.970 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 13 2024-04-28T12:48:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fca580a3bb0 msgr2=0x7fca580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fca580a3bb0 0x7fca580a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009830 tx=0x7fca5c030450 comp rx=0 tx=0).stop 2024-04-28T12:48:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68102a60 msgr2=0x7fca6819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68102a60 0x7fca6819d900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fca5400ea60 tx=0x7fca5400ef20 comp rx=0 tx=0).stop 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 shutdown_connections 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fca580a3bb0 0x7fca580a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca68103420 0x7fca6819ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca68102a60 0x7fca6819d900 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 --2- 172.21.15.53:0/1163815164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fca68101870 0x7fca6806a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 >> 172.21.15.53:0/1163815164 conn(0x7fca680fcde0 msgr2=0x7fca68105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.975+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 shutdown_connections 2024-04-28T12:48:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:40.976+0000 7fca6ee37640 1 -- 172.21.15.53:0/1163815164 wait complete. 2024-04-28T12:48:40.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:41.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 -- 172.21.15.53:0/3141929089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 msgr2=0x7f39d8101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 --2- 172.21.15.53:0/3141929089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d8101c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009900 tx=0x7f39c002a410 comp rx=0 tx=0).stop 2024-04-28T12:48:41.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 -- 172.21.15.53:0/3141929089 shutdown_connections 2024-04-28T12:48:41.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 --2- 172.21.15.53:0/3141929089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39d8103420 0x7f39d8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 --2- 172.21.15.53:0/3141929089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d8102a60 0x7f39d8102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 --2- 172.21.15.53:0/3141929089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d8101c90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 -- 172.21.15.53:0/3141929089 >> 172.21.15.53:0/3141929089 conn(0x7f39d80fce20 msgr2=0x7f39d80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 -- 172.21.15.53:0/3141929089 shutdown_connections 2024-04-28T12:48:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.090+0000 7f39dd778640 1 -- 172.21.15.53:0/3141929089 wait complete. 2024-04-28T12:48:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 Processor -- start 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 -- start start 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d8102a60 0x7f39d819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39d8103420 0x7f39d819ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f39d810d370 con 0x7f39d8101870 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f39d810d1f0 con 0x7f39d8103420 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39dd778640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f39d810d4f0 con 0x7f39d8102a60 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39d6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39d6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d806bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57310/0 (socket says 172.21.15.53:57310) 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39d6ffd640 1 -- 172.21.15.53:0/2783425893 learned_addr learned my addr 172.21.15.53:0/2783425893 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39d77fe640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39d8103420 0x7f39d819ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.091+0000 7f39d67fc640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d8102a60 0x7f39d819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39d77fe640 1 -- 172.21.15.53:0/2783425893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d8102a60 msgr2=0x7f39d819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39d77fe640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d8102a60 0x7f39d819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39d77fe640 1 -- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 msgr2=0x7f39d806bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39d77fe640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d806bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39d77fe640 1 -- 172.21.15.53:0/2783425893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d81a0820 con 0x7f39d8103420 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39d6ffd640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d806bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39d77fe640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39d8103420 0x7f39d819ff10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009fd0 tx=0x7f39d000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39cffff640 1 -- 172.21.15.53:0/2783425893 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d000ce20 con 0x7f39d8103420 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d81a0ab0 con 0x7f39d8103420 2024-04-28T12:48:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39cffff640 1 -- 172.21.15.53:0/2783425893 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39d0005020 con 0x7f39d8103420 2024-04-28T12:48:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39cffff640 1 -- 172.21.15.53:0/2783425893 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0010690 con 0x7f39d8103420 2024-04-28T12:48:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.092+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d81a1000 con 0x7f39d8103420 2024-04-28T12:48:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.093+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d8108cb0 con 0x7f39d8103420 2024-04-28T12:48:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.094+0000 7f39cffff640 1 -- 172.21.15.53:0/2783425893 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f39d00054d0 con 0x7f39d8103420 2024-04-28T12:48:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.095+0000 7f39cffff640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f39c40a3bb0 0x7f39c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.095+0000 7f39d6ffd640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f39c40a3bb0 0x7f39c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.095+0000 7f39cffff640 1 -- 172.21.15.53:0/2783425893 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f39d00d0a20 con 0x7f39d8103420 2024-04-28T12:48:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.095+0000 7f39d6ffd640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f39c40a3bb0 0x7f39c40a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f39c0032fd0 tx=0x7f39c00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.098+0000 7f39cffff640 1 -- 172.21.15.53:0/2783425893 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f39d009d8a0 con 0x7f39d8103420 2024-04-28T12:48:41.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.292+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39d802dca0 con 0x7f39d8103420 2024-04-28T12:48:41.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.293+0000 7f39cffff640 1 -- 172.21.15.53:0/2783425893 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18421 (secure 0 0 0) 0x7f39d009d1a0 con 0x7f39d8103420 2024-04-28T12:48:41.289 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:41.289 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:47:51.666968+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:41.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f39c40a3bb0 msgr2=0x7f39c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f39c40a3bb0 0x7f39c40a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f39c0032fd0 tx=0x7f39c00314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39d8103420 msgr2=0x7f39d819ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39d8103420 0x7f39d819ff10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009fd0 tx=0x7f39d000ef10 comp rx=0 tx=0).stop 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 shutdown_connections 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f39c40a3bb0 0x7f39c40a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39d8103420 0x7f39d819ff10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d8102a60 0x7f39d819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 --2- 172.21.15.53:0/2783425893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39d8101870 0x7f39d806bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 >> 172.21.15.53:0/2783425893 conn(0x7f39d80fce20 msgr2=0x7f39d8105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.295+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 shutdown_connections 2024-04-28T12:48:41.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.296+0000 7f39dd778640 1 -- 172.21.15.53:0/2783425893 wait complete. 2024-04-28T12:48:41.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:48:41.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:48:41.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 -- 172.21.15.53:0/3553126857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 msgr2=0x7f5ca0108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/3553126857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca0108130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009900 tx=0x7f5c8802a410 comp rx=0 tx=0).stop 2024-04-28T12:48:41.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 -- 172.21.15.53:0/3553126857 shutdown_connections 2024-04-28T12:48:41.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/3553126857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ca0108670 0x7f5ca010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/3553126857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca0108130 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/3553126857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ca0103410 0x7f5ca0105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 -- 172.21.15.53:0/3553126857 >> 172.21.15.53:0/3553126857 conn(0x7f5ca00fce20 msgr2=0x7f5ca00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 -- 172.21.15.53:0/3553126857 shutdown_connections 2024-04-28T12:48:41.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.432+0000 7f5ca50c5640 1 -- 172.21.15.53:0/3553126857 wait complete. 2024-04-28T12:48:41.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 Processor -- start 2024-04-28T12:48:41.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 -- start start 2024-04-28T12:48:41.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ca0103410 0x7f5ca019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ca0108670 0x7f5ca01a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca010e4b0 con 0x7f5ca0105d40 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca010e330 con 0x7f5ca0108670 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5ca50c5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca010e630 con 0x7f5ca0103410 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5c9659c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca019b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57326/0 (socket says 172.21.15.53:57326) 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5c9659c640 1 -- 172.21.15.53:0/535219335 learned_addr learned my addr 172.21.15.53:0/535219335 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5c9f59e640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ca0108670 0x7f5ca01a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5c9ed9d640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ca0103410 0x7f5ca019ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.433+0000 7f5c9659c640 1 -- 172.21.15.53:0/535219335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ca0103410 msgr2=0x7f5ca019ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9659c640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ca0103410 0x7f5ca019ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9659c640 1 -- 172.21.15.53:0/535219335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ca0108670 msgr2=0x7f5ca01a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9659c640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ca0108670 0x7f5ca01a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9659c640 1 -- 172.21.15.53:0/535219335 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca01a2a30 con 0x7f5ca0105d40 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9659c640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca019b350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c88006010 tx=0x7f5c880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9cd99640 1 -- 172.21.15.53:0/535219335 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8803f030 con 0x7f5ca0105d40 2024-04-28T12:48:41.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9cd99640 1 -- 172.21.15.53:0/535219335 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c88003cb0 con 0x7f5ca0105d40 2024-04-28T12:48:41.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca01a2cc0 con 0x7f5ca0105d40 2024-04-28T12:48:41.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5c9cd99640 1 -- 172.21.15.53:0/535219335 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88035070 con 0x7f5ca0105d40 2024-04-28T12:48:41.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.434+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca01a31b0 con 0x7f5ca0105d40 2024-04-28T12:48:41.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.435+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ca010a1c0 con 0x7f5ca0105d40 2024-04-28T12:48:41.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.436+0000 7f5c9cd99640 1 -- 172.21.15.53:0/535219335 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5c88042030 con 0x7f5ca0105d40 2024-04-28T12:48:41.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.437+0000 7f5c9cd99640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c8c0a3bb0 0x7f5c8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.437+0000 7f5c9ed9d640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c8c0a3bb0 0x7f5c8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.437+0000 7f5c9cd99640 1 -- 172.21.15.53:0/535219335 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5c880f0920 con 0x7f5ca0105d40 2024-04-28T12:48:41.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.437+0000 7f5c9ed9d640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c8c0a3bb0 0x7f5c8c0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c900058c0 tx=0x7f5c9000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.440+0000 7f5c9cd99640 1 -- 172.21.15.53:0/535219335 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c880bd0b0 con 0x7f5ca0105d40 2024-04-28T12:48:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.626+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5ca002dd20 con 0x7f5ca0105d40 2024-04-28T12:48:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.626+0000 7f5c9cd99640 1 -- 172.21.15.53:0/535219335 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f5c880c13b0 con 0x7f5ca0105d40 2024-04-28T12:48:41.623 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T12:48:41.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.628+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c8c0a3bb0 msgr2=0x7f5c8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.628+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c8c0a3bb0 0x7f5c8c0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c900058c0 tx=0x7f5c9000a2d0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.628+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 msgr2=0x7f5ca019b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.628+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca019b350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c88006010 tx=0x7f5c880042f0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 shutdown_connections 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c8c0a3bb0 0x7f5c8c0a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ca0108670 0x7f5ca01a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca0105d40 0x7f5ca019b350 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 --2- 172.21.15.53:0/535219335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ca0103410 0x7f5ca019ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 >> 172.21.15.53:0/535219335 conn(0x7f5ca00fce20 msgr2=0x7f5ca0102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 shutdown_connections 2024-04-28T12:48:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.629+0000 7f5ca50c5640 1 -- 172.21.15.53:0/535219335 wait complete. 2024-04-28T12:48:41.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:41.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1163815164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:41 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2783425893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-28T12:48:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 -- 172.21.15.53:0/1613507469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c102560 msgr2=0x7fab6c1029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 --2- 172.21.15.53:0/1613507469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c102560 0x7fab6c1029e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fab64009ca0 tx=0x7fab6402a950 comp rx=0 tx=0).stop 2024-04-28T12:48:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 -- 172.21.15.53:0/1613507469 shutdown_connections 2024-04-28T12:48:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 --2- 172.21.15.53:0/1613507469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c102560 0x7fab6c1029e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 --2- 172.21.15.53:0/1613507469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab6c101bc0 0x7fab6c102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 --2- 172.21.15.53:0/1613507469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab6c101260 0x7fab6c101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 -- 172.21.15.53:0/1613507469 >> 172.21.15.53:0/1613507469 conn(0x7fab6c0fcde0 msgr2=0x7fab6c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 -- 172.21.15.53:0/1613507469 shutdown_connections 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.745+0000 7fab746f8640 1 -- 172.21.15.53:0/1613507469 wait complete. 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 Processor -- start 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 -- start start 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab6c101260 0x7fab6c19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c101bc0 0x7fab6c19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab6c102560 0x7fab6c1a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1106a0 con 0x7fab6c101bc0 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c110520 con 0x7fab6c101260 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab746f8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c110820 con 0x7fab6c102560 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c101bc0 0x7fab6c19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c101bc0 0x7fab6c19b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57344/0 (socket says 172.21.15.53:57344) 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 -- 172.21.15.53:0/3578092259 learned_addr learned my addr 172.21.15.53:0/3578092259 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab72c95640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab6c102560 0x7fab6c1a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab72494640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab6c101260 0x7fab6c19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 -- 172.21.15.53:0/3578092259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab6c102560 msgr2=0x7fab6c1a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab6c102560 0x7fab6c1a2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 -- 172.21.15.53:0/3578092259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab6c101260 msgr2=0x7fab6c19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab6c101260 0x7fab6c19ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 -- 172.21.15.53:0/3578092259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab6c1a2910 con 0x7fab6c101bc0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.746+0000 7fab71c93640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c101bc0 0x7fab6c19b240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c920 tx=0x7fab6800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.747+0000 7fab637fe640 1 -- 172.21.15.53:0/3578092259 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68007c70 con 0x7fab6c101bc0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.747+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab6c1a2c00 con 0x7fab6c101bc0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.747+0000 7fab637fe640 1 -- 172.21.15.53:0/3578092259 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab68007e10 con 0x7fab6c101bc0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.747+0000 7fab637fe640 1 -- 172.21.15.53:0/3578092259 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68018820 con 0x7fab6c101bc0 2024-04-28T12:48:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.747+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab6c1a31a0 con 0x7fab6c101bc0 2024-04-28T12:48:41.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.748+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab6c10c150 con 0x7fab6c101bc0 2024-04-28T12:48:41.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.749+0000 7fab637fe640 1 -- 172.21.15.53:0/3578092259 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fab6800f450 con 0x7fab6c101bc0 2024-04-28T12:48:41.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.750+0000 7fab637fe640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab580a3bb0 0x7fab580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.750+0000 7fab637fe640 1 -- 172.21.15.53:0/3578092259 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fab680d0350 con 0x7fab6c101bc0 2024-04-28T12:48:41.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.750+0000 7fab72494640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab580a3bb0 0x7fab580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.750+0000 7fab72494640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab580a3bb0 0x7fab580a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab6c19c1d0 tx=0x7fab5c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.754+0000 7fab637fe640 1 -- 172.21.15.53:0/3578092259 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fab68014030 con 0x7fab6c101bc0 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:41.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1163815164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:41 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2783425893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:41.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.832 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1163815164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.833 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:41 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2783425893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.944+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fab6c02dad0 con 0x7fab6c101bc0 2024-04-28T12:48:41.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.944+0000 7fab637fe640 1 -- 172.21.15.53:0/3578092259 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fab6809d1d0 con 0x7fab6c101bc0 2024-04-28T12:48:41.940 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:41.940 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-04-28T12:48:41.940 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-04-28T12:48:41.941 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-04-28T12:48:41.941 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-04-28T12:48:41.941 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-28T12:48:41.941 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-04-28T12:48:41.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab580a3bb0 msgr2=0x7fab580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab580a3bb0 0x7fab580a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab6c19c1d0 tx=0x7fab5c00b390 comp rx=0 tx=0).stop 2024-04-28T12:48:41.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c101bc0 msgr2=0x7fab6c19b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c101bc0 0x7fab6c19b240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c920 tx=0x7fab6800cde0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 shutdown_connections 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab580a3bb0 0x7fab580a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fab6c102560 0x7fab6c1a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab6c101bc0 0x7fab6c19b240 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 --2- 172.21.15.53:0/3578092259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab6c101260 0x7fab6c19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 >> 172.21.15.53:0/3578092259 conn(0x7fab6c0fcde0 msgr2=0x7fab6c0fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 shutdown_connections 2024-04-28T12:48:41.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:41.946+0000 7fab746f8640 1 -- 172.21.15.53:0/3578092259 wait complete. 2024-04-28T12:48:41.953 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-28T12:48:41.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-28T12:48:42.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 -- 172.21.15.53:0/2897569685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8108650 msgr2=0x7fa1b810aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2897569685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8108650 0x7fa1b810aaa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009ca0 tx=0x7fa1b002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 -- 172.21.15.53:0/2897569685 shutdown_connections 2024-04-28T12:48:42.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2897569685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8108650 0x7fa1b810aaa0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2897569685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b8105d20 0x7fa1b8108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2897569685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1b81033f0 0x7fa1b81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 -- 172.21.15.53:0/2897569685 >> 172.21.15.53:0/2897569685 conn(0x7fa1b80fcde0 msgr2=0x7fa1b80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 -- 172.21.15.53:0/2897569685 shutdown_connections 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.063+0000 7fa1bd589640 1 -- 172.21.15.53:0/2897569685 wait complete. 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 Processor -- start 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 -- start start 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b81033f0 0x7fa1b8102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8105d20 0x7fa1b81013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1b8108650 0x7fa1b81018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b810e4c0 con 0x7fa1b8105d20 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b810e340 con 0x7fa1b81033f0 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1bd589640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b810e640 con 0x7fa1b8108650 2024-04-28T12:48:42.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1b67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8105d20 0x7fa1b81013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1b67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8105d20 0x7fa1b81013a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57362/0 (socket says 172.21.15.53:57362) 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1b67fc640 1 -- 172.21.15.53:0/2841188930 learned_addr learned my addr 172.21.15.53:0/2841188930 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.064+0000 7fa1b6ffd640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b81033f0 0x7fa1b8102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1b77fe640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1b8108650 0x7fa1b81018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1b67fc640 1 -- 172.21.15.53:0/2841188930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1b8108650 msgr2=0x7fa1b81018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1b67fc640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1b8108650 0x7fa1b81018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1b67fc640 1 -- 172.21.15.53:0/2841188930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b81033f0 msgr2=0x7fa1b8102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1b67fc640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b81033f0 0x7fa1b8102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1b67fc640 1 -- 172.21.15.53:0/2841188930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b8102170 con 0x7fa1b8105d20 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1b67fc640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8105d20 0x7fa1b81013a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800e9c0 tx=0x7fa1a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1affff640 1 -- 172.21.15.53:0/2841188930 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a800cd80 con 0x7fa1b8105d20 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1affff640 1 -- 172.21.15.53:0/2841188930 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1a80054d0 con 0x7fa1b8105d20 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b81a2cf0 con 0x7fa1b8105d20 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1affff640 1 -- 172.21.15.53:0/2841188930 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8010430 con 0x7fa1b8105d20 2024-04-28T12:48:42.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.065+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b81a31d0 con 0x7fa1b8105d20 2024-04-28T12:48:42.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.066+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b80fe8d0 con 0x7fa1b8105d20 2024-04-28T12:48:42.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.067+0000 7fa1affff640 1 -- 172.21.15.53:0/2841188930 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa1a8004fd0 con 0x7fa1b8105d20 2024-04-28T12:48:42.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.068+0000 7fa1affff640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa1a40a3c30 0x7fa1a40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.068+0000 7fa1b6ffd640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa1a40a3c30 0x7fa1a40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.068+0000 7fa1affff640 1 -- 172.21.15.53:0/2841188930 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa1a80d0320 con 0x7fa1b8105d20 2024-04-28T12:48:42.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.068+0000 7fa1b6ffd640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa1a40a3c30 0x7fa1a40a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009bd0 tx=0x7fa1a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.072+0000 7fa1affff640 1 -- 172.21.15.53:0/2841188930 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa1a809d1a0 con 0x7fa1b8105d20 2024-04-28T12:48:42.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.280+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa1b802dad0 con 0x7fa1b8105d20 2024-04-28T12:48:42.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.490+0000 7fa1affff640 1 -- 172.21.15.53:0/2841188930 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7fa1a809caa0 con 0x7fa1b8105d20 2024-04-28T12:48:42.486 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-04-28T12:48:42.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.492+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa1a40a3c30 msgr2=0x7fa1a40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.492+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa1a40a3c30 0x7fa1a40a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009bd0 tx=0x7fa1a0005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:42.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.492+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8105d20 msgr2=0x7fa1b81013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.492+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8105d20 0x7fa1b81013a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800e9c0 tx=0x7fa1a800ee80 comp rx=0 tx=0).stop 2024-04-28T12:48:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.492+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 shutdown_connections 2024-04-28T12:48:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.492+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa1a40a3c30 0x7fa1a40a60f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.492+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1b8108650 0x7fa1b81018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.493+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b8105d20 0x7fa1b81013a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.493+0000 7fa1bd589640 1 --2- 172.21.15.53:0/2841188930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b81033f0 0x7fa1b8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.493+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 >> 172.21.15.53:0/2841188930 conn(0x7fa1b80fcde0 msgr2=0x7fa1b8103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.494+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 shutdown_connections 2024-04-28T12:48:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.495+0000 7fa1bd589640 1 -- 172.21.15.53:0/2841188930 wait complete. 2024-04-28T12:48:42.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.626+0000 7f16ebc93640 1 -- 172.21.15.53:0/3317686407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 msgr2=0x7f16e4109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.626+0000 7f16ebc93640 1 --2- 172.21.15.53:0/3317686407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 0x7f16e4109c90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009ca0 tx=0x7f16dc02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- 172.21.15.53:0/3317686407 shutdown_connections 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 --2- 172.21.15.53:0/3317686407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 0x7f16e4109c90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 --2- 172.21.15.53:0/3317686407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4102a40 0x7f16e4102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 --2- 172.21.15.53:0/3317686407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4101850 0x7f16e4101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- 172.21.15.53:0/3317686407 >> 172.21.15.53:0/3317686407 conn(0x7f16e40fcdc0 msgr2=0x7f16e40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- 172.21.15.53:0/3317686407 shutdown_connections 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- 172.21.15.53:0/3317686407 wait complete. 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 Processor -- start 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- start start 2024-04-28T12:48:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4101850 0x7f16e406a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4102a40 0x7f16e419d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 0x7f16e419fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16e410d3b0 con 0x7f16e4103400 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f16e410d230 con 0x7f16e4102a40 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.627+0000 7f16ebc93640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f16e410d530 con 0x7f16e4101850 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16ea230640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 0x7f16e419fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4102a40 0x7f16e419d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4102a40 0x7f16e419d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.53:33056/0 (socket says 172.21.15.53:33056) 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16ea230640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 0x7f16e419fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57392/0 (socket says 172.21.15.53:57392) 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 -- 172.21.15.53:0/363434361 learned_addr learned my addr 172.21.15.53:0/363434361 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e9a2f640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4101850 0x7f16e406a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 -- 172.21.15.53:0/363434361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4101850 msgr2=0x7f16e406a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4101850 0x7f16e406a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 -- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 msgr2=0x7f16e419fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 0x7f16e419fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 -- 172.21.15.53:0/363434361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e41a07a0 con 0x7f16e4102a40 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e9a2f640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4101850 0x7f16e406a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16e922e640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4102a40 0x7f16e419d870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f16d400d950 tx=0x7f16d400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16daffd640 1 -- 172.21.15.53:0/363434361 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4017070 con 0x7f16e4102a40 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e41a0a30 con 0x7f16e4102a40 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.628+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e41a0fd0 con 0x7f16e4102a40 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.629+0000 7f16daffd640 1 -- 172.21.15.53:0/363434361 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16d4009c30 con 0x7f16e4102a40 2024-04-28T12:48:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.629+0000 7f16daffd640 1 -- 172.21.15.53:0/363434361 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4020060 con 0x7f16e4102a40 2024-04-28T12:48:42.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.629+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c80052c0 con 0x7f16e4102a40 2024-04-28T12:48:42.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.630+0000 7f16daffd640 1 -- 172.21.15.53:0/363434361 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f16d4010460 con 0x7f16e4102a40 2024-04-28T12:48:42.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.631+0000 7f16daffd640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f16d00a3bb0 0x7f16d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.631+0000 7f16e9a2f640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f16d00a3bb0 0x7f16d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.631+0000 7f16daffd640 1 -- 172.21.15.53:0/363434361 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f16d40cf920 con 0x7f16e4102a40 2024-04-28T12:48:42.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.632+0000 7f16e9a2f640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f16d00a3bb0 0x7f16d00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f16e41028a0 tx=0x7f16e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.636+0000 7f16daffd640 1 -- 172.21.15.53:0/363434361 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f16d4009de0 con 0x7f16e4102a40 2024-04-28T12:48:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:42 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/535219335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:42 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3578092259' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T12:48:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:42 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2841188930' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:42 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:42 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:42 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:42 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:42.827+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f16c80055b0 con 0x7f16e4102a40 2024-04-28T12:48:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:42 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/535219335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:42 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3578092259' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T12:48:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:42 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2841188930' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:42 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:42 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:42 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:42 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:42 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/535219335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:42.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:42 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3578092259' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T12:48:42.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:42 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2841188930' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:42.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:42 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:42 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:42 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:42.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:42 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.498+0000 7f16daffd640 1 -- 172.21.15.53:0/363434361 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f16d409c590 con 0x7f16e4102a40 2024-04-28T12:48:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f16d00a3bb0 msgr2=0x7f16d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f16d00a3bb0 0x7f16d00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f16e41028a0 tx=0x7f16e0005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4102a40 msgr2=0x7f16e419d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4102a40 0x7f16e419d870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f16d400d950 tx=0x7f16d400de10 comp rx=0 tx=0).stop 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 shutdown_connections 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f16d00a3bb0 0x7f16d00a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e4103400 0x7f16e419fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16e4102a40 0x7f16e419d870 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 --2- 172.21.15.53:0/363434361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4101850 0x7f16e406a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 >> 172.21.15.53:0/363434361 conn(0x7f16e40fcdc0 msgr2=0x7f16e4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 shutdown_connections 2024-04-28T12:48:43.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.503+0000 7f16ebc93640 1 -- 172.21.15.53:0/363434361 wait complete. 2024-04-28T12:48:43.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 -- 172.21.15.53:0/2075209651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 msgr2=0x7fdc1010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 --2- 172.21.15.53:0/2075209651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc1010add0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009ca0 tx=0x7fdc0002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 -- 172.21.15.53:0/2075209651 shutdown_connections 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 --2- 172.21.15.53:0/2075209651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc1010add0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 --2- 172.21.15.53:0/2075209651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc10103b80 0x7fdc10104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 --2- 172.21.15.53:0/2075209651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc10102990 0x7fdc10102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 -- 172.21.15.53:0/2075209651 >> 172.21.15.53:0/2075209651 conn(0x7fdc100fdf00 msgr2=0x7fdc10100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 -- 172.21.15.53:0/2075209651 shutdown_connections 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 -- 172.21.15.53:0/2075209651 wait complete. 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.637+0000 7fdc14a86640 1 Processor -- start 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc14a86640 1 -- start start 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc14a86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc10102990 0x7fdc10198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc14a86640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc10103b80 0x7fdc101990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc14a86640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc101a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc14a86640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1010e4c0 con 0x7fdc10104540 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc14a86640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1010e340 con 0x7fdc10102990 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc14a86640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1010e640 con 0x7fdc10103b80 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc101a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc101a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57406/0 (socket says 172.21.15.53:57406) 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 -- 172.21.15.53:0/1357247523 learned_addr learned my addr 172.21.15.53:0/1357247523 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0dd9b640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc10103b80 0x7fdc101990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0e59c640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc10102990 0x7fdc10198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 -- 172.21.15.53:0/1357247523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc10103b80 msgr2=0x7fdc101990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc10103b80 0x7fdc101990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 -- 172.21.15.53:0/1357247523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc10102990 msgr2=0x7fdc10198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc10102990 0x7fdc10198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 -- 172.21.15.53:0/1357247523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc101a07a0 con 0x7fdc10104540 2024-04-28T12:48:43.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0e59c640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc10102990 0x7fdc10198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:43.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.638+0000 7fdc0ed9d640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc101a0130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdc000097b0 tx=0x7fdc00031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.639+0000 7fdc077fe640 1 -- 172.21.15.53:0/1357247523 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc00003b70 con 0x7fdc10104540 2024-04-28T12:48:43.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.639+0000 7fdc077fe640 1 -- 172.21.15.53:0/1357247523 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc00003d10 con 0x7fdc10104540 2024-04-28T12:48:43.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.639+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc101a0a30 con 0x7fdc10104540 2024-04-28T12:48:43.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.639+0000 7fdc077fe640 1 -- 172.21.15.53:0/1357247523 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0002fc10 con 0x7fdc10104540 2024-04-28T12:48:43.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.639+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc101a0f80 con 0x7fdc10104540 2024-04-28T12:48:43.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.640+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc10109d80 con 0x7fdc10104540 2024-04-28T12:48:43.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.641+0000 7fdc077fe640 1 -- 172.21.15.53:0/1357247523 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdc00004040 con 0x7fdc10104540 2024-04-28T12:48:43.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.641+0000 7fdc077fe640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdbfc0a3bb0 0x7fdbfc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.642+0000 7fdc0e59c640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdbfc0a3bb0 0x7fdbfc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.642+0000 7fdc077fe640 1 -- 172.21.15.53:0/1357247523 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7fdc000f1a00 con 0x7fdc10104540 2024-04-28T12:48:43.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.642+0000 7fdc0e59c640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdbfc0a3bb0 0x7fdbfc0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc101039e0 tx=0x7fdc08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.645+0000 7fdc077fe640 1 -- 172.21.15.53:0/1357247523 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdc000be590 con 0x7fdc10104540 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2841188930' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/363434361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:43.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:43 smithi191 ceph-mon[33016]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:43.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:43.872+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdc100635a0 con 0x7fdc10104540 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2841188930' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/363434361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:43.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:43 smithi053 ceph-mon[29182]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2841188930' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/363434361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:43.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:43 smithi077 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:44.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.510+0000 7fdc077fe640 1 -- 172.21.15.53:0/1357247523 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fdc000bde90 con 0x7fdc10104540 2024-04-28T12:48:44.507 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 2 and data pool 3 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdbfc0a3bb0 msgr2=0x7fdbfc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdbfc0a3bb0 0x7fdbfc0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc101039e0 tx=0x7fdc08005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 msgr2=0x7fdc101a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc101a0130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdc000097b0 tx=0x7fdc00031770 comp rx=0 tx=0).stop 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 shutdown_connections 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdbfc0a3bb0 0x7fdbfc0a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc10104540 0x7fdc101a0130 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc10103b80 0x7fdc101990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 --2- 172.21.15.53:0/1357247523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc10102990 0x7fdc10198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 >> 172.21.15.53:0/1357247523 conn(0x7fdc100fdf00 msgr2=0x7fdc10106f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:44.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 shutdown_connections 2024-04-28T12:48:44.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.513+0000 7fdc14a86640 1 -- 172.21.15.53:0/1357247523 wait complete. 2024-04-28T12:48:44.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:44.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 -- 172.21.15.53:0/522303696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 msgr2=0x7f9e7c102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 --2- 172.21.15.53:0/522303696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c102f00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009900 tx=0x7f9e6c02a410 comp rx=0 tx=0).stop 2024-04-28T12:48:44.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 -- 172.21.15.53:0/522303696 shutdown_connections 2024-04-28T12:48:44.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 --2- 172.21.15.53:0/522303696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e7c103440 0x7f9e7c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 --2- 172.21.15.53:0/522303696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c102f00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 --2- 172.21.15.53:0/522303696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e7c101890 0x7f9e7c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 -- 172.21.15.53:0/522303696 >> 172.21.15.53:0/522303696 conn(0x7f9e7c0fce20 msgr2=0x7f9e7c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 -- 172.21.15.53:0/522303696 shutdown_connections 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.633+0000 7f9e83620640 1 -- 172.21.15.53:0/522303696 wait complete. 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 Processor -- start 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 -- start start 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e7c101890 0x7f9e7c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e7c103440 0x7f9e7c1a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c10e4e0 con 0x7f9e7c102a80 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c10e360 con 0x7f9e7c103440 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e83620640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c10e660 con 0x7f9e7c101890 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e80bbb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e80bbb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57426/0 (socket says 172.21.15.53:57426) 2024-04-28T12:48:44.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e80bbb640 1 -- 172.21.15.53:0/452044089 learned_addr learned my addr 172.21.15.53:0/452044089 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e813bc640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e7c101890 0x7f9e7c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e81bbd640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e7c103440 0x7f9e7c1a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.634+0000 7f9e80bbb640 1 -- 172.21.15.53:0/452044089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e7c101890 msgr2=0x7f9e7c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e80bbb640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e7c101890 0x7f9e7c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e80bbb640 1 -- 172.21.15.53:0/452044089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e7c103440 msgr2=0x7f9e7c1a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e80bbb640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e7c103440 0x7f9e7c1a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e80bbb640 1 -- 172.21.15.53:0/452044089 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7c1a07e0 con 0x7f9e7c102a80 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e80bbb640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c199100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c005940 tx=0x7f9e6c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e727fc640 1 -- 172.21.15.53:0/452044089 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c03f030 con 0x7f9e7c102a80 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e727fc640 1 -- 172.21.15.53:0/452044089 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e6c003aa0 con 0x7f9e7c102a80 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e727fc640 1 -- 172.21.15.53:0/452044089 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c035070 con 0x7f9e7c102a80 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7c1a0a70 con 0x7f9e7c102a80 2024-04-28T12:48:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.635+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e7c1a0f60 con 0x7f9e7c102a80 2024-04-28T12:48:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.636+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e7c108c30 con 0x7f9e7c102a80 2024-04-28T12:48:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.638+0000 7f9e727fc640 1 -- 172.21.15.53:0/452044089 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9e6c042030 con 0x7f9e7c102a80 2024-04-28T12:48:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.638+0000 7f9e727fc640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e680a3bb0 0x7f9e680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.639+0000 7f9e813bc640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e680a3bb0 0x7f9e680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.639+0000 7f9e727fc640 1 -- 172.21.15.53:0/452044089 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9e6c0f0ec0 con 0x7f9e7c102a80 2024-04-28T12:48:44.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.639+0000 7f9e813bc640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e680a3bb0 0x7f9e680a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c1028e0 tx=0x7f9e7800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:44.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.642+0000 7f9e727fc640 1 -- 172.21.15.53:0/452044089 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9e6c0bd300 con 0x7f9e7c102a80 2024-04-28T12:48:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:44 smithi191 ceph-mon[33016]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T12:48:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:44 smithi191 ceph-mon[33016]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T12:48:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:44 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1357247523' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:44 smithi191 ceph-mon[33016]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-28T12:48:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:44 smithi191 ceph-mon[33016]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T12:48:44.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.831+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e7c02dca0 con 0x7f9e7c102a80 2024-04-28T12:48:44.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.833+0000 7f9e727fc640 1 -- 172.21.15.53:0/452044089 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22197 (secure 0 0 0) 0x7f9e6c0c1600 con 0x7f9e7c102a80 2024-04-28T12:48:44.829 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:44.829 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":85,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:48:43.873840+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e680a3bb0 msgr2=0x7f9e680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e680a3bb0 0x7f9e680a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c1028e0 tx=0x7f9e7800b390 comp rx=0 tx=0).stop 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 msgr2=0x7f9e7c199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c199100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c005940 tx=0x7f9e6c031a50 comp rx=0 tx=0).stop 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 shutdown_connections 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9e680a3bb0 0x7f9e680a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e7c103440 0x7f9e7c1a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7c102a80 0x7f9e7c199100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 --2- 172.21.15.53:0/452044089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e7c101890 0x7f9e7c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 >> 172.21.15.53:0/452044089 conn(0x7f9e7c0fce20 msgr2=0x7f9e7c105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:44.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 shutdown_connections 2024-04-28T12:48:44.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.835+0000 7f9e83620640 1 -- 172.21.15.53:0/452044089 wait complete. 2024-04-28T12:48:44.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:44 smithi053 ceph-mon[29182]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T12:48:44.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:44 smithi053 ceph-mon[29182]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T12:48:44.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:44 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1357247523' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:44.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:44 smithi053 ceph-mon[29182]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-28T12:48:44.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:44 smithi053 ceph-mon[29182]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T12:48:44.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:44.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 -- 172.21.15.53:0/3138846203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8064104550 msgr2=0x7f806410ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 --2- 172.21.15.53:0/3138846203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8064104550 0x7f806410ade0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8054009ca0 tx=0x7f805402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 -- 172.21.15.53:0/3138846203 shutdown_connections 2024-04-28T12:48:44.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 --2- 172.21.15.53:0/3138846203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8064104550 0x7f806410ade0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 --2- 172.21.15.53:0/3138846203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8064103b90 0x7f8064104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 --2- 172.21.15.53:0/3138846203 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80641029a0 0x7f8064102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 -- 172.21.15.53:0/3138846203 >> 172.21.15.53:0/3138846203 conn(0x7f80640fdf30 msgr2=0x7f8064100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 -- 172.21.15.53:0/3138846203 shutdown_connections 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 -- 172.21.15.53:0/3138846203 wait complete. 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 Processor -- start 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.949+0000 7f8069293640 1 -- start start 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80641029a0 0x7f8064198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8064103b90 0x7f80641990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8064104550 0x7f80641a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f806410e4d0 con 0x7f80641029a0 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f806410e350 con 0x7f8064104550 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f806410e650 con 0x7f8064103b90 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80641029a0 0x7f8064198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80641029a0 0x7f8064198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57438/0 (socket says 172.21.15.53:57438) 2024-04-28T12:48:44.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 -- 172.21.15.53:0/3159394712 learned_addr learned my addr 172.21.15.53:0/3159394712 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f80637fe640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8064104550 0x7f80641a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f80627fc640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8064103b90 0x7f80641990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 -- 172.21.15.53:0/3159394712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8064103b90 msgr2=0x7f80641990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8064103b90 0x7f80641990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 -- 172.21.15.53:0/3159394712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8064104550 msgr2=0x7f80641a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8064104550 0x7f80641a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 -- 172.21.15.53:0/3159394712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80641a0790 con 0x7f80641029a0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8062ffd640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80641029a0 0x7f8064198b70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f805c00d950 tx=0x7f805c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f804bfff640 1 -- 172.21.15.53:0/3159394712 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c017070 con 0x7f80641029a0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f804bfff640 1 -- 172.21.15.53:0/3159394712 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f805c009c30 con 0x7f80641029a0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80641a0a80 con 0x7f80641029a0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f804bfff640 1 -- 172.21.15.53:0/3159394712 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c00f620 con 0x7f80641029a0 2024-04-28T12:48:44.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.950+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80641a0f80 con 0x7f80641029a0 2024-04-28T12:48:44.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:44 smithi077 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T12:48:44.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:44 smithi077 ceph-mon[32553]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T12:48:44.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:44 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1357247523' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:44.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:44 smithi077 ceph-mon[32553]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-28T12:48:44.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:44 smithi077 ceph-mon[32553]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T12:48:44.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.952+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8064109d90 con 0x7f80641029a0 2024-04-28T12:48:44.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.953+0000 7f804bfff640 1 -- 172.21.15.53:0/3159394712 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f805c01a070 con 0x7f80641029a0 2024-04-28T12:48:44.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.953+0000 7f804bfff640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f804c0a3bb0 0x7f804c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:44.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.954+0000 7f80627fc640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f804c0a3bb0 0x7f804c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:44.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.954+0000 7f804bfff640 1 -- 172.21.15.53:0/3159394712 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f805c0cfda0 con 0x7f80641029a0 2024-04-28T12:48:44.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.954+0000 7f80627fc640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f804c0a3bb0 0x7f804c0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f806419a080 tx=0x7f8058005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:44.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:44.958+0000 7f804bfff640 1 -- 172.21.15.53:0/3159394712 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f805c009de0 con 0x7f80641029a0 2024-04-28T12:48:45.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.144+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f806402dd50 con 0x7f80641029a0 2024-04-28T12:48:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.146+0000 7f804bfff640 1 -- 172.21.15.53:0/3159394712 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22197 (secure 0 0 0) 0x7f805c09c8d0 con 0x7f80641029a0 2024-04-28T12:48:45.142 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:45.142 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":85,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:48:43.873840+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:45.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.147+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f804c0a3bb0 msgr2=0x7f804c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.147+0000 7f8069293640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f804c0a3bb0 0x7f804c0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f806419a080 tx=0x7f8058005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80641029a0 msgr2=0x7f8064198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80641029a0 0x7f8064198b70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f805c00d950 tx=0x7f805c00de10 comp rx=0 tx=0).stop 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 shutdown_connections 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f804c0a3bb0 0x7f804c0a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8064104550 0x7f80641a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8064103b90 0x7f80641990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 --2- 172.21.15.53:0/3159394712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80641029a0 0x7f8064198b70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 >> 172.21.15.53:0/3159394712 conn(0x7f80640fdf30 msgr2=0x7f8064106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 shutdown_connections 2024-04-28T12:48:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.148+0000 7f8069293640 1 -- 172.21.15.53:0/3159394712 wait complete. 2024-04-28T12:48:45.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-28T12:48:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 -- 172.21.15.53:0/3700539832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14108670 msgr2=0x7fec1410aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 --2- 172.21.15.53:0/3700539832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14108670 0x7fec1410aac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009ca0 tx=0x7fec0c02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 -- 172.21.15.53:0/3700539832 shutdown_connections 2024-04-28T12:48:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 --2- 172.21.15.53:0/3700539832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14108670 0x7fec1410aac0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 --2- 172.21.15.53:0/3700539832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec14105d40 0x7fec14108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 --2- 172.21.15.53:0/3700539832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec14103410 0x7fec14105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 -- 172.21.15.53:0/3700539832 >> 172.21.15.53:0/3700539832 conn(0x7fec140fce20 msgr2=0x7fec140ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 -- 172.21.15.53:0/3700539832 shutdown_connections 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 -- 172.21.15.53:0/3700539832 wait complete. 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 Processor -- start 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.265+0000 7fec1a366640 1 -- start start 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14103410 0x7fec14198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec14105d40 0x7fec14199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec14108670 0x7fec141a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fec1410e530 con 0x7fec14103410 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec1410e3b0 con 0x7fec14108670 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fec1410e6b0 con 0x7fec14105d40 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14103410 0x7fec14198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14103410 0x7fec14198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57460/0 (socket says 172.21.15.53:57460) 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 -- 172.21.15.53:0/3168482465 learned_addr learned my addr 172.21.15.53:0/3168482465 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec137fe640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec14105d40 0x7fec14199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 -- 172.21.15.53:0/3168482465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec14105d40 msgr2=0x7fec14199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec14105d40 0x7fec14199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 -- 172.21.15.53:0/3168482465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec14108670 msgr2=0x7fec141a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec18903640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec14108670 0x7fec141a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec14108670 0x7fec141a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 -- 172.21.15.53:0/3168482465 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec141a0830 con 0x7fec14103410 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec13fff640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14103410 0x7fec14198c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fec0400b4a0 tx=0x7fec0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec117fa640 1 -- 172.21.15.53:0/3168482465 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04014070 con 0x7fec14103410 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec117fa640 1 -- 172.21.15.53:0/3168482465 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec04010ac0 con 0x7fec14103410 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec141a0b20 con 0x7fec14103410 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec117fa640 1 -- 172.21.15.53:0/3168482465 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04017c30 con 0x7fec14103410 2024-04-28T12:48:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.266+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec141a1070 con 0x7fec14103410 2024-04-28T12:48:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.267+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec140fe910 con 0x7fec14103410 2024-04-28T12:48:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.269+0000 7fec117fa640 1 -- 172.21.15.53:0/3168482465 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fec04017070 con 0x7fec14103410 2024-04-28T12:48:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.269+0000 7fec117fa640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7febfc0a3bb0 0x7febfc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.269+0000 7fec137fe640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7febfc0a3bb0 0x7febfc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.269+0000 7fec117fa640 1 -- 172.21.15.53:0/3168482465 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fec040cfb20 con 0x7fec14103410 2024-04-28T12:48:45.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.270+0000 7fec137fe640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7febfc0a3bb0 0x7febfc0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec08005fd0 tx=0x7fec08005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:45.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.273+0000 7fec117fa640 1 -- 172.21.15.53:0/3168482465 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fec040cfe20 con 0x7fec14103410 2024-04-28T12:48:45.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:45.521+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fec1402dad0 con 0x7fec14103410 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: pgmap v289: 97 pgs: 6 creating+peering, 7 active+clean, 84 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1357247523' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/452044089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3159394712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:45 smithi191 ceph-mon[33016]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T12:48:45.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: pgmap v289: 97 pgs: 6 creating+peering, 7 active+clean, 84 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1357247523' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T12:48:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-28T12:48:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-28T12:48:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/452044089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3159394712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:45.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:45 smithi053 ceph-mon[29182]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T12:48:45.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: pgmap v289: 97 pgs: 6 creating+peering, 7 active+clean, 84 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:45.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1357247523' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T12:48:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-28T12:48:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-28T12:48:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/452044089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3159394712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:45 smithi077 ceph-mon[32553]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T12:48:46.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.525+0000 7fec117fa640 1 -- 172.21.15.53:0/3168482465 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fec0409c650 con 0x7fec14103410 2024-04-28T12:48:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7febfc0a3bb0 msgr2=0x7febfc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7febfc0a3bb0 0x7febfc0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec08005fd0 tx=0x7fec08005850 comp rx=0 tx=0).stop 2024-04-28T12:48:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14103410 msgr2=0x7fec14198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14103410 0x7fec14198c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fec0400b4a0 tx=0x7fec0400b960 comp rx=0 tx=0).stop 2024-04-28T12:48:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 shutdown_connections 2024-04-28T12:48:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7febfc0a3bb0 0x7febfc0a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec14108670 0x7fec141a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec14105d40 0x7fec14199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 --2- 172.21.15.53:0/3168482465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec14103410 0x7fec14198c20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 >> 172.21.15.53:0/3168482465 conn(0x7fec140fce20 msgr2=0x7fec14103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:46.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 shutdown_connections 2024-04-28T12:48:46.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.528+0000 7fec1a366640 1 -- 172.21.15.53:0/3168482465 wait complete. 2024-04-28T12:48:46.538 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-28T12:48:46.538 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.653+0000 7fb55d10a640 1 -- 172.21.15.53:0/521082559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 msgr2=0x7fb558109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.653+0000 7fb55d10a640 1 --2- 172.21.15.53:0/521082559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb558109cd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb548009ca0 tx=0x7fb54802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.653+0000 7fb55d10a640 1 -- 172.21.15.53:0/521082559 shutdown_connections 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.653+0000 7fb55d10a640 1 --2- 172.21.15.53:0/521082559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb558109cd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.653+0000 7fb55d10a640 1 --2- 172.21.15.53:0/521082559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb558102a80 0x7fb558102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.653+0000 7fb55d10a640 1 --2- 172.21.15.53:0/521082559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb558101890 0x7fb558101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.653+0000 7fb55d10a640 1 -- 172.21.15.53:0/521082559 >> 172.21.15.53:0/521082559 conn(0x7fb5580fce20 msgr2=0x7fb5580ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.654+0000 7fb55d10a640 1 -- 172.21.15.53:0/521082559 shutdown_connections 2024-04-28T12:48:46.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.654+0000 7fb55d10a640 1 -- 172.21.15.53:0/521082559 wait complete. 2024-04-28T12:48:46.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.654+0000 7fb55d10a640 1 Processor -- start 2024-04-28T12:48:46.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.654+0000 7fb55d10a640 1 -- start start 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb558101890 0x7fb558198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb558102a80 0x7fb558199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb5581a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb55810d3c0 con 0x7fb558103440 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb55810d240 con 0x7fb558101890 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb55810d540 con 0x7fb558102a80 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb5581a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb5581a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38494/0 (socket says 172.21.15.53:38494) 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 -- 172.21.15.53:0/751137126 learned_addr learned my addr 172.21.15.53:0/751137126 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb556d9d640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb558101890 0x7fb558198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:46.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55659c640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb558102a80 0x7fb558199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 -- 172.21.15.53:0/751137126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb558102a80 msgr2=0x7fb558199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb558102a80 0x7fb558199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 -- 172.21.15.53:0/751137126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb558101890 msgr2=0x7fb558198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb558101890 0x7fb558198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 -- 172.21.15.53:0/751137126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5581a0800 con 0x7fb558103440 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55759e640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb5581a0190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097b0 tx=0x7fb548003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb54ffff640 1 -- 172.21.15.53:0/751137126 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548004240 con 0x7fb558103440 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5581a0a90 con 0x7fb558103440 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb54ffff640 1 -- 172.21.15.53:0/751137126 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5480317c0 con 0x7fb558103440 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb54ffff640 1 -- 172.21.15.53:0/751137126 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54802fd80 con 0x7fb558103440 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.655+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5581a1030 con 0x7fb558103440 2024-04-28T12:48:46.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.656+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5581a1200 con 0x7fb558103440 2024-04-28T12:48:46.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.658+0000 7fb54ffff640 1 -- 172.21.15.53:0/751137126 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb5480043e0 con 0x7fb558103440 2024-04-28T12:48:46.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.658+0000 7fb54ffff640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb5440a3bb0 0x7fb5440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:46.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.658+0000 7fb556d9d640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb5440a3bb0 0x7fb5440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:46.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.658+0000 7fb54ffff640 1 -- 172.21.15.53:0/751137126 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb5480f1da0 con 0x7fb558103440 2024-04-28T12:48:46.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.659+0000 7fb556d9d640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb5440a3bb0 0x7fb5440a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb550005d00 tx=0x7fb550005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:46.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.662+0000 7fb54ffff640 1 -- 172.21.15.53:0/751137126 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb548080060 con 0x7fb558103440 2024-04-28T12:48:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:46 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] up:active 2024-04-28T12:48:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:46 smithi191 ceph-mon[33016]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:46 smithi191 ceph-mon[33016]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:46 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3168482465' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T12:48:46.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:46.889+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fb5581a14f0 con 0x7fb558103440 2024-04-28T12:48:46.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:46 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] up:active 2024-04-28T12:48:46.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:46 smithi053 ceph-mon[29182]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:46.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:46 smithi053 ceph-mon[29182]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:46.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:46 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3168482465' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T12:48:46.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:46 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] up:active 2024-04-28T12:48:46.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:46 smithi077 ceph-mon[32553]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:46.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:46 smithi077 ceph-mon[32553]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:46.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:46 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3168482465' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T12:48:47.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.526+0000 7fb54ffff640 1 -- 172.21.15.53:0/751137126 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fb5480be9f0 con 0x7fb558103440 2024-04-28T12:48:47.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.529+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb5440a3bb0 msgr2=0x7fb5440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.529+0000 7fb55d10a640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb5440a3bb0 0x7fb5440a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb550005d00 tx=0x7fb550005c50 comp rx=0 tx=0).stop 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.529+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 msgr2=0x7fb5581a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.529+0000 7fb55d10a640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb5581a0190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097b0 tx=0x7fb548003c20 comp rx=0 tx=0).stop 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.529+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 shutdown_connections 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.529+0000 7fb55d10a640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb5440a3bb0 0x7fb5440a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.530+0000 7fb55d10a640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb558103440 0x7fb5581a0190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.530+0000 7fb55d10a640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb558102a80 0x7fb558199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.530+0000 7fb55d10a640 1 --2- 172.21.15.53:0/751137126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb558101890 0x7fb558198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.530+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 >> 172.21.15.53:0/751137126 conn(0x7fb5580fce20 msgr2=0x7fb5580fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.530+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 shutdown_connections 2024-04-28T12:48:47.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.530+0000 7fb55d10a640 1 -- 172.21.15.53:0/751137126 wait complete. 2024-04-28T12:48:47.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 -- 172.21.15.53:0/3198896795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528103b80 msgr2=0x7f0528104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 --2- 172.21.15.53:0/3198896795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528103b80 0x7f0528104000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f051c009900 tx=0x7f051c02a410 comp rx=0 tx=0).stop 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 -- 172.21.15.53:0/3198896795 shutdown_connections 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 --2- 172.21.15.53:0/3198896795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0528104540 0x7f052810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 --2- 172.21.15.53:0/3198896795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528103b80 0x7f0528104000 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 --2- 172.21.15.53:0/3198896795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0528102990 0x7f0528102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 -- 172.21.15.53:0/3198896795 >> 172.21.15.53:0/3198896795 conn(0x7f05280fdf00 msgr2=0x7f0528100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 -- 172.21.15.53:0/3198896795 shutdown_connections 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 -- 172.21.15.53:0/3198896795 wait complete. 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 Processor -- start 2024-04-28T12:48:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.659+0000 7f052e4da640 1 -- start start 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f052e4da640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528102990 0x7f0528198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f052e4da640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0528103b80 0x7f0528199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f052e4da640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0528104540 0x7f05281a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f052e4da640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f052810e4f0 con 0x7f0528102990 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f052e4da640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f052810e370 con 0x7f0528104540 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f052e4da640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f052810e670 con 0x7f0528103b80 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528102990 0x7f0528198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528102990 0x7f0528198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38500/0 (socket says 172.21.15.53:38500) 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 -- 172.21.15.53:0/783722020 learned_addr learned my addr 172.21.15.53:0/783722020 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f052ca77640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0528104540 0x7f05281a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f05277fe640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0528103b80 0x7f0528199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 -- 172.21.15.53:0/783722020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0528103b80 msgr2=0x7f0528199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0528103b80 0x7f0528199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 -- 172.21.15.53:0/783722020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0528104540 msgr2=0x7f05281a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0528104540 0x7f05281a0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 -- 172.21.15.53:0/783722020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05281a0700 con 0x7f0528102990 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f05277fe640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0528103b80 0x7f0528199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.660+0000 7f0527fff640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528102990 0x7f0528198ae0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f051800c970 tx=0x7f051800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.661+0000 7f05257fa640 1 -- 172.21.15.53:0/783722020 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0518007c10 con 0x7f0528102990 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.661+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05281a09f0 con 0x7f0528102990 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.661+0000 7f05257fa640 1 -- 172.21.15.53:0/783722020 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0518007db0 con 0x7f0528102990 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.661+0000 7f05257fa640 1 -- 172.21.15.53:0/783722020 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051800f7d0 con 0x7f0528102990 2024-04-28T12:48:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.661+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05281a0f60 con 0x7f0528102990 2024-04-28T12:48:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.662+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0528069ba0 con 0x7f0528102990 2024-04-28T12:48:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.663+0000 7f05257fa640 1 -- 172.21.15.53:0/783722020 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0518018480 con 0x7f0528102990 2024-04-28T12:48:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.664+0000 7f05257fa640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05100a3bb0 0x7f05100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.664+0000 7f05277fe640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05100a3bb0 0x7f05100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.664+0000 7f05257fa640 1 -- 172.21.15.53:0/783722020 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f05180d06b0 con 0x7f0528102990 2024-04-28T12:48:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.664+0000 7f05277fe640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05100a3bb0 0x7f05100a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c0314c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.668+0000 7f05257fa640 1 -- 172.21.15.53:0/783722020 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f051809d1d0 con 0x7f0528102990 2024-04-28T12:48:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:47 smithi191 ceph-mon[33016]: pgmap v292: 97 pgs: 11 creating+peering, 56 active+clean, 30 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2024-04-28T12:48:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:47 smithi191 ceph-mon[33016]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-28T12:48:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:47 smithi191 ceph-mon[33016]: Cluster is now healthy 2024-04-28T12:48:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:47 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3168482465' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T12:48:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:47 smithi191 ceph-mon[33016]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:47.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:47 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/751137126' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:48:47.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:47.894+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f0528109cb0 con 0x7f0528102990 2024-04-28T12:48:47.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:47 smithi053 ceph-mon[29182]: pgmap v292: 97 pgs: 11 creating+peering, 56 active+clean, 30 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2024-04-28T12:48:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:47 smithi053 ceph-mon[29182]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-28T12:48:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:47 smithi053 ceph-mon[29182]: Cluster is now healthy 2024-04-28T12:48:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3168482465' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T12:48:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:47 smithi053 ceph-mon[29182]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/751137126' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:48:47.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:47 smithi077 ceph-mon[32553]: pgmap v292: 97 pgs: 11 creating+peering, 56 active+clean, 30 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2024-04-28T12:48:47.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:47 smithi077 ceph-mon[32553]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-28T12:48:47.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:47 smithi077 ceph-mon[32553]: Cluster is now healthy 2024-04-28T12:48:47.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:47 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3168482465' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T12:48:47.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:47 smithi077 ceph-mon[32553]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:47.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:47 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/751137126' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:48:48.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.536+0000 7f05257fa640 1 -- 172.21.15.53:0/783722020 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f0528109cb0 con 0x7f0528102990 2024-04-28T12:48:48.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05100a3bb0 msgr2=0x7f05100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:48.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05100a3bb0 0x7f05100a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c0314c0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528102990 msgr2=0x7f0528198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:48.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528102990 0x7f0528198ae0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f051800c970 tx=0x7f051800ce30 comp rx=0 tx=0).stop 2024-04-28T12:48:48.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 shutdown_connections 2024-04-28T12:48:48.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05100a3bb0 0x7f05100a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0528104540 0x7f05281a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0528103b80 0x7f0528199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 --2- 172.21.15.53:0/783722020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0528102990 0x7f0528198ae0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 >> 172.21.15.53:0/783722020 conn(0x7f05280fdf00 msgr2=0x7f0528106e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:48.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 shutdown_connections 2024-04-28T12:48:48.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.539+0000 7f052e4da640 1 -- 172.21.15.53:0/783722020 wait complete. 2024-04-28T12:48:48.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-28T12:48:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 -- 172.21.15.53:0/1024175889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0103410 msgr2=0x7f99b0105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 --2- 172.21.15.53:0/1024175889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0103410 0x7f99b0105800 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009900 tx=0x7f99a802a410 comp rx=0 tx=0).stop 2024-04-28T12:48:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 -- 172.21.15.53:0/1024175889 shutdown_connections 2024-04-28T12:48:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 --2- 172.21.15.53:0/1024175889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b0108670 0x7f99b010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 --2- 172.21.15.53:0/1024175889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b0105d40 0x7f99b0108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 --2- 172.21.15.53:0/1024175889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0103410 0x7f99b0105800 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 -- 172.21.15.53:0/1024175889 >> 172.21.15.53:0/1024175889 conn(0x7f99b00fce20 msgr2=0x7f99b00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 -- 172.21.15.53:0/1024175889 shutdown_connections 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.668+0000 7f99b73af640 1 -- 172.21.15.53:0/1024175889 wait complete. 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 Processor -- start 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 -- start start 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b0103410 0x7f99b0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b0105d40 0x7f99b0199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0108670 0x7f99b01a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f99b010e4c0 con 0x7f99b0108670 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99b010e340 con 0x7f99b0103410 2024-04-28T12:48:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b73af640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f99b010e640 con 0x7f99b0105d40 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b594c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0108670 0x7f99b01a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b514b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b0103410 0x7f99b0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b594c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0108670 0x7f99b01a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38528/0 (socket says 172.21.15.53:38528) 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b594c640 1 -- 172.21.15.53:0/4242376534 learned_addr learned my addr 172.21.15.53:0/4242376534 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.669+0000 7f99b494a640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b0105d40 0x7f99b0199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b594c640 1 -- 172.21.15.53:0/4242376534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b0105d40 msgr2=0x7f99b0199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b594c640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b0105d40 0x7f99b0199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b594c640 1 -- 172.21.15.53:0/4242376534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b0103410 msgr2=0x7f99b0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b594c640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b0103410 0x7f99b0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b594c640 1 -- 172.21.15.53:0/4242376534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b01a0830 con 0x7f99b0108670 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b594c640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0108670 0x7f99b01a0190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00eaa0 tx=0x7f99ac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99a67fc640 1 -- 172.21.15.53:0/4242376534 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac00cdb0 con 0x7f99b0108670 2024-04-28T12:48:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99a67fc640 1 -- 172.21.15.53:0/4242376534 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99ac005020 con 0x7f99b0108670 2024-04-28T12:48:48.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99a67fc640 1 -- 172.21.15.53:0/4242376534 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac0116d0 con 0x7f99b0108670 2024-04-28T12:48:48.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b01a0b20 con 0x7f99b0108670 2024-04-28T12:48:48.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.670+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b01a1020 con 0x7f99b0108670 2024-04-28T12:48:48.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.671+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b00fe910 con 0x7f99b0108670 2024-04-28T12:48:48.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.672+0000 7f99a67fc640 1 -- 172.21.15.53:0/4242376534 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f99ac0051d0 con 0x7f99b0108670 2024-04-28T12:48:48.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.673+0000 7f99a67fc640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f999c0a3bb0 0x7f999c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:48.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.673+0000 7f99b514b640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f999c0a3bb0 0x7f999c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:48.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.673+0000 7f99a67fc640 1 -- 172.21.15.53:0/4242376534 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f99ac0d0530 con 0x7f99b0108670 2024-04-28T12:48:48.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.674+0000 7f99b514b640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f999c0a3bb0 0x7f999c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009230 tx=0x7f99a80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:48.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.678+0000 7f99a67fc640 1 -- 172.21.15.53:0/4242376534 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f99ac016030 con 0x7f99b0108670 2024-04-28T12:48:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:48 smithi191 ceph-mon[33016]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T12:48:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:48 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/751137126' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T12:48:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:48 smithi191 ceph-mon[33016]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T12:48:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:48 smithi191 ceph-mon[33016]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:48 smithi191 ceph-mon[33016]: fsmap cephfs:2 {0=i=up:active,1=h=up:creating} 10 up:standby 2024-04-28T12:48:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:48 smithi191 ceph-mon[33016]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-28T12:48:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:48 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/783722020' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:48:48.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:48 smithi077 ceph-mon[32553]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T12:48:48.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:48 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/751137126' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T12:48:48.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:48 smithi077 ceph-mon[32553]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T12:48:48.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:48 smithi077 ceph-mon[32553]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:48.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:48 smithi077 ceph-mon[32553]: fsmap cephfs:2 {0=i=up:active,1=h=up:creating} 10 up:standby 2024-04-28T12:48:48.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:48 smithi077 ceph-mon[32553]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-28T12:48:48.850 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:48 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/783722020' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:48:48.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:48 smithi053 ceph-mon[29182]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T12:48:48.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:48 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/751137126' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T12:48:48.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:48 smithi053 ceph-mon[29182]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T12:48:48.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:48 smithi053 ceph-mon[29182]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-28T12:48:48.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:48 smithi053 ceph-mon[29182]: fsmap cephfs:2 {0=i=up:active,1=h=up:creating} 10 up:standby 2024-04-28T12:48:48.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:48 smithi053 ceph-mon[29182]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-28T12:48:48.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:48 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/783722020' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:48:48.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:48.906+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f99b0063530 con 0x7f99b0108670 2024-04-28T12:48:49.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.456+0000 7f99a67fc640 1 -- 172.21.15.53:0/4242376534 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f99ac09c970 con 0x7f99b0108670 2024-04-28T12:48:49.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f999c0a3bb0 msgr2=0x7f999c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f999c0a3bb0 0x7f999c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009230 tx=0x7f99a80314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0108670 msgr2=0x7f99b01a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0108670 0x7f99b01a0190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00eaa0 tx=0x7f99ac00ef60 comp rx=0 tx=0).stop 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 shutdown_connections 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f999c0a3bb0 0x7f999c0a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b0108670 0x7f99b01a0190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b0105d40 0x7f99b0199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 --2- 172.21.15.53:0/4242376534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b0103410 0x7f99b0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 >> 172.21.15.53:0/4242376534 conn(0x7f99b00fce20 msgr2=0x7f99b0105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 shutdown_connections 2024-04-28T12:48:49.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.459+0000 7f99b73af640 1 -- 172.21.15.53:0/4242376534 wait complete. 2024-04-28T12:48:49.469 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-28T12:48:49.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-28T12:48:49.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 -- 172.21.15.53:0/3250510132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 msgr2=0x7f066c105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 --2- 172.21.15.53:0/3250510132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c105840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0668009900 tx=0x7f066802a410 comp rx=0 tx=0).stop 2024-04-28T12:48:49.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 -- 172.21.15.53:0/3250510132 shutdown_connections 2024-04-28T12:48:49.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 --2- 172.21.15.53:0/3250510132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c1086b0 0x7f066c10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 --2- 172.21.15.53:0/3250510132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066c105d80 0x7f066c108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 --2- 172.21.15.53:0/3250510132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c105840 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 -- 172.21.15.53:0/3250510132 >> 172.21.15.53:0/3250510132 conn(0x7f066c0fcde0 msgr2=0x7f066c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 -- 172.21.15.53:0/3250510132 shutdown_connections 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.586+0000 7f06736f9640 1 -- 172.21.15.53:0/3250510132 wait complete. 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 Processor -- start 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 -- start start 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066c105d80 0x7f066c1990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c1086b0 0x7f066c1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f066c10e4c0 con 0x7f066c103450 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f066c10e340 con 0x7f066c105d80 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f06736f9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f066c10e640 con 0x7f066c1086b0 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:49.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38538/0 (socket says 172.21.15.53:38538) 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 -- 172.21.15.53:0/68776960 learned_addr learned my addr 172.21.15.53:0/68776960 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0670c94640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066c105d80 0x7f066c1990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671c96640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c1086b0 0x7f066c1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 -- 172.21.15.53:0/68776960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c1086b0 msgr2=0x7f066c1a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c1086b0 0x7f066c1a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 -- 172.21.15.53:0/68776960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066c105d80 msgr2=0x7f066c1990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066c105d80 0x7f066c1990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.587+0000 7f0671495640 1 -- 172.21.15.53:0/68776960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066c1a07b0 con 0x7f066c103450 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.588+0000 7f0671495640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c198b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0668005940 tx=0x7f0668004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.588+0000 7f06627fc640 1 -- 172.21.15.53:0/68776960 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066803f030 con 0x7f066c103450 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.588+0000 7f06627fc640 1 -- 172.21.15.53:0/68776960 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0668003cb0 con 0x7f066c103450 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.588+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066c1a0a40 con 0x7f066c103450 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.588+0000 7f06627fc640 1 -- 172.21.15.53:0/68776960 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668035070 con 0x7f066c103450 2024-04-28T12:48:49.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.588+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066c1a0f30 con 0x7f066c103450 2024-04-28T12:48:49.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.589+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f066c0fe8d0 con 0x7f066c103450 2024-04-28T12:48:49.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.590+0000 7f06627fc640 1 -- 172.21.15.53:0/68776960 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0668042030 con 0x7f066c103450 2024-04-28T12:48:49.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.590+0000 7f06627fc640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f06580a3bb0 0x7f06580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.591+0000 7f06627fc640 1 -- 172.21.15.53:0/68776960 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f06680f0e00 con 0x7f066c103450 2024-04-28T12:48:49.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.591+0000 7f0670c94640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f06580a3bb0 0x7f06580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:49.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.591+0000 7f0670c94640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f06580a3bb0 0x7f06580a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f066c19a0a0 tx=0x7f065c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:49.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.594+0000 7f06627fc640 1 -- 172.21.15.53:0/68776960 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f06680bd930 con 0x7f066c103450 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: pgmap v293: 97 pgs: 11 creating+peering, 86 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/783722020' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] up:active 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: Cluster is now healthy 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: fsmap cephfs:2 {0=i=up:active,1=h=up:active} 10 up:standby 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:creating} 9 up:standby 2024-04-28T12:48:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-28T12:48:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4242376534' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:48:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4242376534' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T12:48:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] up:active 2024-04-28T12:48:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:49 smithi191 ceph-mon[33016]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:48:49.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.807+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f066c062d70 con 0x7f06580a3bb0 2024-04-28T12:48:49.808 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: pgmap v293: 97 pgs: 11 creating+peering, 86 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-28T12:48:49.808 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/783722020' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T12:48:49.808 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] up:active 2024-04-28T12:48:49.808 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: Cluster is now healthy 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: fsmap cephfs:2 {0=i=up:active,1=h=up:active} 10 up:standby 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:creating} 9 up:standby 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4242376534' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4242376534' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] up:active 2024-04-28T12:48:49.809 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-mon[29182]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:48:49.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.827+0000 7f06627fc640 1 -- 172.21.15.53:0/68776960 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f066c062d70 con 0x7f06580a3bb0 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f06580a3bb0 msgr2=0x7f06580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f06580a3bb0 0x7f06580a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f066c19a0a0 tx=0x7f065c00a2d0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 msgr2=0x7f066c198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c198b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0668005940 tx=0x7f0668004430 comp rx=0 tx=0).stop 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 shutdown_connections 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f06580a3bb0 0x7f06580a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c1086b0 0x7f066c1a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066c105d80 0x7f066c1990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 --2- 172.21.15.53:0/68776960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f066c103450 0x7f066c198b90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 >> 172.21.15.53:0/68776960 conn(0x7f066c0fcde0 msgr2=0x7f066c103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 shutdown_connections 2024-04-28T12:48:49.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.829+0000 7f06736f9640 1 -- 172.21.15.53:0/68776960 wait complete. 2024-04-28T12:48:49.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-28T12:48:49.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: pgmap v293: 97 pgs: 11 creating+peering, 86 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-28T12:48:49.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/783722020' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T12:48:49.866 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] up:active 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: Cluster is now healthy 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: fsmap cephfs:2 {0=i=up:active,1=h=up:active} 10 up:standby 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:creating} 9 up:standby 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4242376534' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4242376534' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] up:active 2024-04-28T12:48:49.867 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:49 smithi077 ceph-mon[32553]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:48:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 -- 172.21.15.53:0/3180239447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc103b80 msgr2=0x7ffacc104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/3180239447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc103b80 0x7ffacc104000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009900 tx=0x7ffab402a410 comp rx=0 tx=0).stop 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 -- 172.21.15.53:0/3180239447 shutdown_connections 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/3180239447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffacc104540 0x7ffacc10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/3180239447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc103b80 0x7ffacc104000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/3180239447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffacc102990 0x7ffacc102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 -- 172.21.15.53:0/3180239447 >> 172.21.15.53:0/3180239447 conn(0x7ffacc0fdf00 msgr2=0x7ffacc100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 -- 172.21.15.53:0/3180239447 shutdown_connections 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.946+0000 7ffad1ed9640 1 -- 172.21.15.53:0/3180239447 wait complete. 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 Processor -- start 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 -- start start 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc102990 0x7ffacc198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffacc103b80 0x7ffacc199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffacc104540 0x7ffacc1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10e4c0 con 0x7ffacc102990 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10e340 con 0x7ffacc103b80 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffad1ed9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10e640 con 0x7ffacc104540 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffacb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc102990 0x7ffacc198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffacb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc102990 0x7ffacc198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38560/0 (socket says 172.21.15.53:38560) 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffacb7fe640 1 -- 172.21.15.53:0/1591228118 learned_addr learned my addr 172.21.15.53:0/1591228118 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.947+0000 7ffacaffd640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffacc103b80 0x7ffacc199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffacb7fe640 1 -- 172.21.15.53:0/1591228118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffacc104540 msgr2=0x7ffacc1a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffacb7fe640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffacc104540 0x7ffacc1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffacb7fe640 1 -- 172.21.15.53:0/1591228118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffacc103b80 msgr2=0x7ffacc199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffacb7fe640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffacc103b80 0x7ffacc199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffacb7fe640 1 -- 172.21.15.53:0/1591228118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc1a0830 con 0x7ffacc102990 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffacb7fe640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc102990 0x7ffacc198c20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ffac400d950 tx=0x7ffac400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffac8ff9640 1 -- 172.21.15.53:0/1591228118 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4017070 con 0x7ffacc102990 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc1a0b20 con 0x7ffacc102990 2024-04-28T12:48:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffac8ff9640 1 -- 172.21.15.53:0/1591228118 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac4009c30 con 0x7ffacc102990 2024-04-28T12:48:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffac8ff9640 1 -- 172.21.15.53:0/1591228118 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac400f620 con 0x7ffacc102990 2024-04-28T12:48:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.948+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc1a1020 con 0x7ffacc102990 2024-04-28T12:48:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.949+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc109dd0 con 0x7ffacc102990 2024-04-28T12:48:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.950+0000 7ffac8ff9640 1 -- 172.21.15.53:0/1591228118 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffac401a070 con 0x7ffacc102990 2024-04-28T12:48:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.951+0000 7ffac8ff9640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffab80a3bb0 0x7ffab80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.951+0000 7ffacaffd640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffab80a3bb0 0x7ffab80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.951+0000 7ffac8ff9640 1 -- 172.21.15.53:0/1591228118 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ffac40cfdb0 con 0x7ffacc102990 2024-04-28T12:48:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.951+0000 7ffacaffd640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffab80a3bb0 0x7ffab80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffab4000c00 tx=0x7ffab40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:49.955+0000 7ffac8ff9640 1 -- 172.21.15.53:0/1591228118 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffac4009de0 con 0x7ffacc102990 2024-04-28T12:48:50.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:48:49.811+0000 7f8514fe2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:48:50.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:48:49.811+0000 7f8514fe2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:48:50.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:48:49.811+0000 7f8514fe2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:48:50.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:48:49.811+0000 7f8514fe2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:48:50.145 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 12:48:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T12:48:49.811+0000 7f8514fe2700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:48:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.170+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ffacc062e00 con 0x7ffab80a3bb0 2024-04-28T12:48:50.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.198+0000 7ffac8ff9640 1 -- 172.21.15.53:0/1591228118 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffacc062e00 con 0x7ffab80a3bb0 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffab80a3bb0 msgr2=0x7ffab80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffab80a3bb0 0x7ffab80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffab4000c00 tx=0x7ffab40314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc102990 msgr2=0x7ffacc198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc102990 0x7ffacc198c20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ffac400d950 tx=0x7ffac400de10 comp rx=0 tx=0).stop 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 shutdown_connections 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ffab80a3bb0 0x7ffab80a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffacc104540 0x7ffacc1a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffacc103b80 0x7ffacc199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 --2- 172.21.15.53:0/1591228118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc102990 0x7ffacc198c20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 >> 172.21.15.53:0/1591228118 conn(0x7ffacc0fdf00 msgr2=0x7ffacc106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 shutdown_connections 2024-04-28T12:48:50.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.200+0000 7ffad1ed9640 1 -- 172.21.15.53:0/1591228118 wait complete. 2024-04-28T12:48:50.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-28T12:48:50.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 -- 172.21.15.53:0/335904181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 msgr2=0x7fc05c109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 --2- 172.21.15.53:0/335904181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c109cd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc054009ca0 tx=0x7fc05402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 -- 172.21.15.53:0/335904181 shutdown_connections 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 --2- 172.21.15.53:0/335904181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c109cd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 --2- 172.21.15.53:0/335904181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05c102a80 0x7fc05c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 --2- 172.21.15.53:0/335904181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c101890 0x7fc05c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 -- 172.21.15.53:0/335904181 >> 172.21.15.53:0/335904181 conn(0x7fc05c0fce20 msgr2=0x7fc05c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 -- 172.21.15.53:0/335904181 shutdown_connections 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 -- 172.21.15.53:0/335904181 wait complete. 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.318+0000 7fc060f51640 1 Processor -- start 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc060f51640 1 -- start start 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc060f51640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c101890 0x7fc05c06bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc060f51640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05c102a80 0x7fc05c19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc060f51640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c19fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc060f51640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c10d3a0 con 0x7fc05c103440 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc060f51640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c10d220 con 0x7fc05c102a80 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc060f51640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c10d520 con 0x7fc05c101890 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c19fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c19fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38576/0 (socket says 172.21.15.53:38576) 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 -- 172.21.15.53:0/2337094802 learned_addr learned my addr 172.21.15.53:0/2337094802 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc059d9b640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05c102a80 0x7fc05c19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05a59c640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c101890 0x7fc05c06bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 -- 172.21.15.53:0/2337094802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c101890 msgr2=0x7fc05c06bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c101890 0x7fc05c06bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 -- 172.21.15.53:0/2337094802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05c102a80 msgr2=0x7fc05c19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05c102a80 0x7fc05c19d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 -- 172.21.15.53:0/2337094802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05c1a0710 con 0x7fc05c103440 2024-04-28T12:48:50.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc059d9b640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05c102a80 0x7fc05c19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.319+0000 7fc05ad9d640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c19fe50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0540097b0 tx=0x7fc054003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.320+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054004240 con 0x7fc05c103440 2024-04-28T12:48:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.320+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0540317c0 con 0x7fc05c103440 2024-04-28T12:48:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.320+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05402fd80 con 0x7fc05c103440 2024-04-28T12:48:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.320+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc05c1a09a0 con 0x7fc05c103440 2024-04-28T12:48:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.320+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc05c1a0ef0 con 0x7fc05c103440 2024-04-28T12:48:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.321+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc05c108bb0 con 0x7fc05c103440 2024-04-28T12:48:50.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.322+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc0540043e0 con 0x7fc05c103440 2024-04-28T12:48:50.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.322+0000 7fc04b7fe640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc0440a3bb0 0x7fc0440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.322+0000 7fc05a59c640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc0440a3bb0 0x7fc0440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.322+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc0540f1cf0 con 0x7fc05c103440 2024-04-28T12:48:50.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.323+0000 7fc05a59c640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc0440a3bb0 0x7fc0440a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0058c0 tx=0x7fc04c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:50.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.326+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc0540be820 con 0x7fc05c103440 2024-04-28T12:48:50.537 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: 2024-04-28T12:48:50.232+0000 7fab9ce23700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-28T12:48:50.537 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: 2024-04-28T12:48:50.232+0000 7fab9ce23700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-28T12:48:50.537 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 12:48:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: 2024-04-28T12:48:50.232+0000 7fab9ce23700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-28T12:48:50.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.545+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fc05c062d70 con 0x7fc0440a3bb0 2024-04-28T12:48:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.554+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc0540ba790 con 0x7fc05c103440 2024-04-28T12:48:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.555+0000 7fc04b7fe640 1 -- 172.21.15.53:0/2337094802 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fc05c062d70 con 0x7fc0440a3bb0 2024-04-28T12:48:50.551 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/qa/sv_0/1a90579e-e6f7-4046-83cf-77f2f5e5bc5d 2024-04-28T12:48:50.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc0440a3bb0 msgr2=0x7fc0440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc0440a3bb0 0x7fc0440a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0058c0 tx=0x7fc04c005850 comp rx=0 tx=0).stop 2024-04-28T12:48:50.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 msgr2=0x7fc05c19fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c19fe50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0540097b0 tx=0x7fc054003c20 comp rx=0 tx=0).stop 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 shutdown_connections 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc0440a3bb0 0x7fc0440a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc05c103440 0x7fc05c19fe50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05c102a80 0x7fc05c19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 --2- 172.21.15.53:0/2337094802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c101890 0x7fc05c06bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 >> 172.21.15.53:0/2337094802 conn(0x7fc05c0fce20 msgr2=0x7fc05c105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 shutdown_connections 2024-04-28T12:48:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.557+0000 7fc060f51640 1 -- 172.21.15.53:0/2337094802 wait complete. 2024-04-28T12:48:50.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-28T12:48:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 -- 172.21.15.53:0/1839691959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac101890 msgr2=0x7f10ac101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 --2- 172.21.15.53:0/1839691959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac101890 0x7f10ac101cb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009900 tx=0x7f10a802a410 comp rx=0 tx=0).stop 2024-04-28T12:48:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 -- 172.21.15.53:0/1839691959 shutdown_connections 2024-04-28T12:48:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 --2- 172.21.15.53:0/1839691959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10ac103440 0x7f10ac109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 --2- 172.21.15.53:0/1839691959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ac102a80 0x7f10ac102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 --2- 172.21.15.53:0/1839691959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac101890 0x7f10ac101cb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 -- 172.21.15.53:0/1839691959 >> 172.21.15.53:0/1839691959 conn(0x7f10ac0fce20 msgr2=0x7f10ac0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 -- 172.21.15.53:0/1839691959 shutdown_connections 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 -- 172.21.15.53:0/1839691959 wait complete. 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 Processor -- start 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.675+0000 7f10b3d66640 1 -- start start 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ac101890 0x7f10ac06bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac102a80 0x7f10ac19d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10ac103440 0x7f10ac19ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac10d3b0 con 0x7f10ac102a80 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac10d230 con 0x7f10ac103440 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac10d530 con 0x7f10ac101890 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac102a80 0x7f10ac19d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac102a80 0x7f10ac19d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38594/0 (socket says 172.21.15.53:38594) 2024-04-28T12:48:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 -- 172.21.15.53:0/2427879748 learned_addr learned my addr 172.21.15.53:0/2427879748 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b2303640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10ac103440 0x7f10ac19ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1b02640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ac101890 0x7f10ac06bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 -- 172.21.15.53:0/2427879748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ac101890 msgr2=0x7f10ac06bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ac101890 0x7f10ac06bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 -- 172.21.15.53:0/2427879748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10ac103440 msgr2=0x7f10ac19ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10ac103440 0x7f10ac19ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 -- 172.21.15.53:0/2427879748 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac1a0840 con 0x7f10ac102a80 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b1301640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac102a80 0x7f10ac19d940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f109c00c970 tx=0x7f109c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10a2ffd640 1 -- 172.21.15.53:0/2427879748 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c007bb0 con 0x7f10ac102a80 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10a2ffd640 1 -- 172.21.15.53:0/2427879748 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109c007d50 con 0x7f10ac102a80 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac1a0b30 con 0x7f10ac102a80 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10a2ffd640 1 -- 172.21.15.53:0/2427879748 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c00f450 con 0x7f10ac102a80 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.676+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac1a1030 con 0x7f10ac102a80 2024-04-28T12:48:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.677+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ac108c50 con 0x7f10ac102a80 2024-04-28T12:48:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.678+0000 7f10a2ffd640 1 -- 172.21.15.53:0/2427879748 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f109c004f20 con 0x7f10ac102a80 2024-04-28T12:48:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.679+0000 7f10a2ffd640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f10980a3c40 0x7f10980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.679+0000 7f10b1b02640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f10980a3c40 0x7f10980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.679+0000 7f10a2ffd640 1 -- 172.21.15.53:0/2427879748 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f109c0d0130 con 0x7f10ac102a80 2024-04-28T12:48:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.680+0000 7f10b1b02640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f10980a3c40 0x7f10980a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10a802a930 tx=0x7f10a8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:50.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.684+0000 7f10a2ffd640 1 -- 172.21.15.53:0/2427879748 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f109c014030 con 0x7f10ac102a80 2024-04-28T12:48:50.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.898+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f10ac062d70 con 0x7f10980a3c40 2024-04-28T12:48:50.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:50 smithi053 ceph-mon[29182]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:50.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:50 smithi053 ceph-mon[29182]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:50.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.931+0000 7f10a2ffd640 1 -- 172.21.15.53:0/2427879748 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f10ac062d70 con 0x7f10980a3c40 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.932+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f10980a3c40 msgr2=0x7f10980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f10980a3c40 0x7f10980a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10a802a930 tx=0x7f10a8000bc0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac102a80 msgr2=0x7f10ac19d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac102a80 0x7f10ac19d940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f109c00c970 tx=0x7f109c00ce30 comp rx=0 tx=0).stop 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 shutdown_connections 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f10980a3c40 0x7f10980a6100 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10ac103440 0x7f10ac19ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10ac102a80 0x7f10ac19d940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 --2- 172.21.15.53:0/2427879748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ac101890 0x7f10ac06bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 >> 172.21.15.53:0/2427879748 conn(0x7f10ac0fce20 msgr2=0x7f10ac105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 shutdown_connections 2024-04-28T12:48:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:50.933+0000 7f10b3d66640 1 -- 172.21.15.53:0/2427879748 wait complete. 2024-04-28T12:48:50.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-28T12:48:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:50 smithi077 ceph-mon[32553]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:50.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:50 smithi077 ceph-mon[32553]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:50 smithi191 ceph-mon[33016]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:50 smithi191 ceph-mon[33016]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.050+0000 7faba8a59640 1 -- 172.21.15.53:0/2603189468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 msgr2=0x7faba410ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 --2- 172.21.15.53:0/2603189468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba410ab00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009ca0 tx=0x7fab9c02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 -- 172.21.15.53:0/2603189468 shutdown_connections 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 --2- 172.21.15.53:0/2603189468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba410ab00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 --2- 172.21.15.53:0/2603189468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba4105de0 0x7faba41081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 --2- 172.21.15.53:0/2603189468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faba41034b0 0x7faba41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 -- 172.21.15.53:0/2603189468 >> 172.21.15.53:0/2603189468 conn(0x7faba40fcde0 msgr2=0x7faba40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 -- 172.21.15.53:0/2603189468 shutdown_connections 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 -- 172.21.15.53:0/2603189468 wait complete. 2024-04-28T12:48:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 Processor -- start 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.051+0000 7faba8a59640 1 -- start start 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba41034b0 0x7faba4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faba4105de0 0x7faba41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba41a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faba410e500 con 0x7faba4108710 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faba410e380 con 0x7faba41034b0 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faba410e680 con 0x7faba4105de0 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba41a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba41a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38616/0 (socket says 172.21.15.53:38616) 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 -- 172.21.15.53:0/840092600 learned_addr learned my addr 172.21.15.53:0/840092600 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba259c640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba41034b0 0x7faba4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba1d9b640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faba4105de0 0x7faba41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 -- 172.21.15.53:0/840092600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faba4105de0 msgr2=0x7faba41990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faba4105de0 0x7faba41990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 -- 172.21.15.53:0/840092600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba41034b0 msgr2=0x7faba4198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba41034b0 0x7faba4198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 -- 172.21.15.53:0/840092600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba41a07d0 con 0x7faba4108710 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba2d9d640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba41a0160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fab9c033040 tx=0x7fab9c0317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7fab937fe640 1 -- 172.21.15.53:0/840092600 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c003b70 con 0x7faba4108710 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7fab937fe640 1 -- 172.21.15.53:0/840092600 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab9c003d10 con 0x7faba4108710 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7fab937fe640 1 -- 172.21.15.53:0/840092600 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c02fbd0 con 0x7faba4108710 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba41a0a60 con 0x7faba4108710 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.052+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba41a1000 con 0x7faba4108710 2024-04-28T12:48:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.053+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba40fe8d0 con 0x7faba4108710 2024-04-28T12:48:51.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.054+0000 7fab937fe640 1 -- 172.21.15.53:0/840092600 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fab9c02fd70 con 0x7faba4108710 2024-04-28T12:48:51.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.055+0000 7fab937fe640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab8c0a3cc0 0x7fab8c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.055+0000 7faba259c640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab8c0a3cc0 0x7fab8c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.055+0000 7fab937fe640 1 -- 172.21.15.53:0/840092600 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fab9c0f1930 con 0x7faba4108710 2024-04-28T12:48:51.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.055+0000 7faba259c640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab8c0a3cc0 0x7fab8c0a6180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fab9400bbe0 tx=0x7fab9400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:51.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.060+0000 7fab937fe640 1 -- 172.21.15.53:0/840092600 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fab9c0be370 con 0x7faba4108710 2024-04-28T12:48:51.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.275+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7faba4062d70 con 0x7fab8c0a3cc0 2024-04-28T12:48:51.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.280+0000 7fab937fe640 1 -- 172.21.15.53:0/840092600 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7faba4062d70 con 0x7fab8c0a3cc0 2024-04-28T12:48:51.276 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/qa/sv_1/068a4084-e7f8-4632-9d3e-602add1a0061 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab8c0a3cc0 msgr2=0x7fab8c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab8c0a3cc0 0x7fab8c0a6180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fab9400bbe0 tx=0x7fab9400b390 comp rx=0 tx=0).stop 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 msgr2=0x7faba41a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba41a0160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fab9c033040 tx=0x7fab9c0317c0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 shutdown_connections 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fab8c0a3cc0 0x7fab8c0a6180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faba4108710 0x7faba41a0160 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faba4105de0 0x7faba41990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 --2- 172.21.15.53:0/840092600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba41034b0 0x7faba4198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 >> 172.21.15.53:0/840092600 conn(0x7faba40fcde0 msgr2=0x7faba4103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:51.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 shutdown_connections 2024-04-28T12:48:51.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.282+0000 7faba8a59640 1 -- 172.21.15.53:0/840092600 wait complete. 2024-04-28T12:48:51.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:51.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 -- 172.21.15.53:0/3692108036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 msgr2=0x7ff67410ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 --2- 172.21.15.53:0/3692108036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff67410ab00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009ca0 tx=0x7ff66c02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 -- 172.21.15.53:0/3692108036 shutdown_connections 2024-04-28T12:48:51.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 --2- 172.21.15.53:0/3692108036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff67410ab00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 --2- 172.21.15.53:0/3692108036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff674105de0 0x7ff6741081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 --2- 172.21.15.53:0/3692108036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6741034b0 0x7ff6741058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 -- 172.21.15.53:0/3692108036 >> 172.21.15.53:0/3692108036 conn(0x7ff6740fcde0 msgr2=0x7ff6740ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:51.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 -- 172.21.15.53:0/3692108036 shutdown_connections 2024-04-28T12:48:51.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.396+0000 7ff679e2c640 1 -- 172.21.15.53:0/3692108036 wait complete. 2024-04-28T12:48:51.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 Processor -- start 2024-04-28T12:48:51.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 -- start start 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6741034b0 0x7ff674198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff674105de0 0x7ff6741990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff6741a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff67410e4b0 con 0x7ff674108710 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff67410e330 con 0x7ff674105de0 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff679e2c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff67410e630 con 0x7ff6741034b0 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff6741a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff6741a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38632/0 (socket says 172.21.15.53:38632) 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 -- 172.21.15.53:0/1689149449 learned_addr learned my addr 172.21.15.53:0/1689149449 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff672ffd640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff674105de0 0x7ff6741990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff6737fe640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6741034b0 0x7ff674198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 -- 172.21.15.53:0/1689149449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6741034b0 msgr2=0x7ff674198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6741034b0 0x7ff674198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 -- 172.21.15.53:0/1689149449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff674105de0 msgr2=0x7ff6741990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff674105de0 0x7ff6741990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.397+0000 7ff673fff640 1 -- 172.21.15.53:0/1689149449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6741a07d0 con 0x7ff674108710 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.398+0000 7ff673fff640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff6741a0160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0097b0 tx=0x7ff66c031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.398+0000 7ff670ff9640 1 -- 172.21.15.53:0/1689149449 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c003bf0 con 0x7ff674108710 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.398+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6741a0a60 con 0x7ff674108710 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.398+0000 7ff670ff9640 1 -- 172.21.15.53:0/1689149449 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff66c003d90 con 0x7ff674108710 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.398+0000 7ff670ff9640 1 -- 172.21.15.53:0/1689149449 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c02fc70 con 0x7ff674108710 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.398+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6741a0fb0 con 0x7ff674108710 2024-04-28T12:48:51.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.399+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6740fe8d0 con 0x7ff674108710 2024-04-28T12:48:51.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.400+0000 7ff670ff9640 1 -- 172.21.15.53:0/1689149449 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff66c004040 con 0x7ff674108710 2024-04-28T12:48:51.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.401+0000 7ff670ff9640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff65c0a3c40 0x7ff65c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.401+0000 7ff670ff9640 1 -- 172.21.15.53:0/1689149449 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff66c0f1ad0 con 0x7ff674108710 2024-04-28T12:48:51.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.401+0000 7ff6737fe640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff65c0a3c40 0x7ff65c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.401+0000 7ff6737fe640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff65c0a3c40 0x7ff65c0a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff6680058c0 tx=0x7ff668005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:51.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.404+0000 7ff670ff9640 1 -- 172.21.15.53:0/1689149449 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff66c0be610 con 0x7ff674108710 2024-04-28T12:48:51.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.634+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff67402dbf0 con 0x7ff674108710 2024-04-28T12:48:51.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.635+0000 7ff670ff9640 1 -- 172.21.15.53:0/1689149449 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11873 (secure 0 0 0) 0x7ff66c0bdf10 con 0x7ff674108710 2024-04-28T12:48:51.631 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:51.631 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:48:49.455677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24485,24478]},"id":1}]} 2024-04-28T12:48:51.631 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 22 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff65c0a3c40 msgr2=0x7ff65c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff65c0a3c40 0x7ff65c0a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff6680058c0 tx=0x7ff668005c60 comp rx=0 tx=0).stop 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 msgr2=0x7ff6741a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff6741a0160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0097b0 tx=0x7ff66c031840 comp rx=0 tx=0).stop 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 shutdown_connections 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff65c0a3c40 0x7ff65c0a6100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff674108710 0x7ff6741a0160 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff674105de0 0x7ff6741990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 --2- 172.21.15.53:0/1689149449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6741034b0 0x7ff674198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 >> 172.21.15.53:0/1689149449 conn(0x7ff6740fcde0 msgr2=0x7ff674103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:51.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 shutdown_connections 2024-04-28T12:48:51.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.637+0000 7ff679e2c640 1 -- 172.21.15.53:0/1689149449 wait complete. 2024-04-28T12:48:51.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 -- 172.21.15.53:0/765864759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378105e00 msgr2=0x7f23781081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 --2- 172.21.15.53:0/765864759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378105e00 0x7f23781081f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f236c009900 tx=0x7f236c02a410 comp rx=0 tx=0).stop 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 -- 172.21.15.53:0/765864759 shutdown_connections 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 --2- 172.21.15.53:0/765864759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2378108730 0x7f237810ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 --2- 172.21.15.53:0/765864759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378105e00 0x7f23781081f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 --2- 172.21.15.53:0/765864759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23781034d0 0x7f23781058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 -- 172.21.15.53:0/765864759 >> 172.21.15.53:0/765864759 conn(0x7f23780fce20 msgr2=0x7f23780ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:51.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 -- 172.21.15.53:0/765864759 shutdown_connections 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.750+0000 7f237e052640 1 -- 172.21.15.53:0/765864759 wait complete. 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 Processor -- start 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 -- start start 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23781034d0 0x7f2378198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2378105e00 0x7f23781991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378108730 0x7f23781a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f237810e4d0 con 0x7f2378108730 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f237810e350 con 0x7f23781034d0 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f237e052640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f237810e650 con 0x7f2378105e00 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f2377fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378108730 0x7f23781a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f2377fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378108730 0x7f23781a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38658/0 (socket says 172.21.15.53:38658) 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f2377fff640 1 -- 172.21.15.53:0/3158531469 learned_addr learned my addr 172.21.15.53:0/3158531469 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.751+0000 7f23777fe640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23781034d0 0x7f2378198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2376ffd640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2378105e00 0x7f23781991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2377fff640 1 -- 172.21.15.53:0/3158531469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2378105e00 msgr2=0x7f23781991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2377fff640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2378105e00 0x7f23781991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2377fff640 1 -- 172.21.15.53:0/3158531469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23781034d0 msgr2=0x7f2378198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2377fff640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23781034d0 0x7f2378198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2377fff640 1 -- 172.21.15.53:0/3158531469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23781a0850 con 0x7f2378108730 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2377fff640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378108730 0x7f23781a01b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f237000eaa0 tx=0x7f237000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2374ff9640 1 -- 172.21.15.53:0/3158531469 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237000cdb0 con 0x7f2378108730 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2374ff9640 1 -- 172.21.15.53:0/3158531469 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2370005020 con 0x7f2378108730 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f2374ff9640 1 -- 172.21.15.53:0/3158531469 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700116d0 con 0x7f2378108730 2024-04-28T12:48:51.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23781a0b40 con 0x7f2378108730 2024-04-28T12:48:51.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.752+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23781a10e0 con 0x7f2378108730 2024-04-28T12:48:51.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.753+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23780fe910 con 0x7f2378108730 2024-04-28T12:48:51.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.754+0000 7f2374ff9640 1 -- 172.21.15.53:0/3158531469 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f23700051d0 con 0x7f2378108730 2024-04-28T12:48:51.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.755+0000 7f2374ff9640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f23600a3cc0 0x7f23600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:51.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.755+0000 7f23777fe640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f23600a3cc0 0x7f23600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:51.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.755+0000 7f2374ff9640 1 -- 172.21.15.53:0/3158531469 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f23700d0610 con 0x7f2378108730 2024-04-28T12:48:51.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.756+0000 7f23777fe640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f23600a3cc0 0x7f23600a6180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f237819a110 tx=0x7f2368005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:51.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.760+0000 7f2374ff9640 1 -- 172.21.15.53:0/3158531469 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2370016030 con 0x7f2378108730 2024-04-28T12:48:51.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:51 smithi053 ceph-mon[29182]: pgmap v294: 97 pgs: 97 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 294 B/s rd, 2.9 KiB/s wr, 12 op/s 2024-04-28T12:48:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:51 smithi053 ceph-mon[29182]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:51 smithi053 ceph-mon[29182]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-28T12:48:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:51 smithi053 ceph-mon[29182]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:51 smithi053 ceph-mon[29182]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.949+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f237802dd30 con 0x7f2378108730 2024-04-28T12:48:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.951+0000 7f2374ff9640 1 -- 172.21.15.53:0/3158531469 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22184 (secure 0 0 0) 0x7f237009c8f0 con 0x7f2378108730 2024-04-28T12:48:51.947 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:51.947 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":86,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:48:45.456811+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:51.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:51 smithi077 ceph-mon[32553]: pgmap v294: 97 pgs: 97 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 294 B/s rd, 2.9 KiB/s wr, 12 op/s 2024-04-28T12:48:51.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:51 smithi077 ceph-mon[32553]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:51 smithi077 ceph-mon[32553]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-28T12:48:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:51 smithi077 ceph-mon[32553]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:51 smithi077 ceph-mon[32553]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f23600a3cc0 msgr2=0x7f23600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f23600a3cc0 0x7f23600a6180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f237819a110 tx=0x7f2368005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378108730 msgr2=0x7f23781a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378108730 0x7f23781a01b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f237000eaa0 tx=0x7f237000ef60 comp rx=0 tx=0).stop 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 shutdown_connections 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f23600a3cc0 0x7f23600a6180 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2378108730 0x7f23781a01b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2378105e00 0x7f23781991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 --2- 172.21.15.53:0/3158531469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23781034d0 0x7f2378198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 >> 172.21.15.53:0/3158531469 conn(0x7f23780fce20 msgr2=0x7f2378103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 shutdown_connections 2024-04-28T12:48:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:51.953+0000 7f237e052640 1 -- 172.21.15.53:0/3158531469 wait complete. 2024-04-28T12:48:51.960 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-28T12:48:51.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-28T12:48:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:51 smithi191 ceph-mon[33016]: pgmap v294: 97 pgs: 97 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 294 B/s rd, 2.9 KiB/s wr, 12 op/s 2024-04-28T12:48:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:51 smithi191 ceph-mon[33016]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:51 smithi191 ceph-mon[33016]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-28T12:48:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:51 smithi191 ceph-mon[33016]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:52.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:51 smithi191 ceph-mon[33016]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:48:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 -- 172.21.15.53:0/2437122744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 msgr2=0x7fcd4c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 --2- 172.21.15.53:0/2437122744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c1057e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009ca0 tx=0x7fcd4402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 -- 172.21.15.53:0/2437122744 shutdown_connections 2024-04-28T12:48:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 --2- 172.21.15.53:0/2437122744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4c108650 0x7fcd4c10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 --2- 172.21.15.53:0/2437122744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4c105d20 0x7fcd4c108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 --2- 172.21.15.53:0/2437122744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c1057e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 -- 172.21.15.53:0/2437122744 >> 172.21.15.53:0/2437122744 conn(0x7fcd4c0fcde0 msgr2=0x7fcd4c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 -- 172.21.15.53:0/2437122744 shutdown_connections 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.066+0000 7fcd518dc640 1 -- 172.21.15.53:0/2437122744 wait complete. 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 Processor -- start 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 -- start start 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4c105d20 0x7fcd4c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4c108650 0x7fcd4c1a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c10e4d0 con 0x7fcd4c1033f0 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c10e350 con 0x7fcd4c108650 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd518dc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c10e650 con 0x7fcd4c105d20 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd4affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd4affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38682/0 (socket says 172.21.15.53:38682) 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.067+0000 7fcd4affd640 1 -- 172.21.15.53:0/1878798344 learned_addr learned my addr 172.21.15.53:0/1878798344 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4b7fe640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4c108650 0x7fcd4c1a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4a7fc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4c105d20 0x7fcd4c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4affd640 1 -- 172.21.15.53:0/1878798344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4c105d20 msgr2=0x7fcd4c1991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4affd640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4c105d20 0x7fcd4c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4affd640 1 -- 172.21.15.53:0/1878798344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4c108650 msgr2=0x7fcd4c1a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4affd640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4c108650 0x7fcd4c1a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4affd640 1 -- 172.21.15.53:0/1878798344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4c1a0870 con 0x7fcd4c1033f0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd4affd640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c198c60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcd44006010 tx=0x7fcd44003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd508da640 1 -- 172.21.15.53:0/1878798344 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4402f380 con 0x7fcd4c1033f0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4c1a0b00 con 0x7fcd4c1033f0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd508da640 1 -- 172.21.15.53:0/1878798344 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd440317c0 con 0x7fcd4c1033f0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd508da640 1 -- 172.21.15.53:0/1878798344 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4402fb90 con 0x7fcd4c1033f0 2024-04-28T12:48:52.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.068+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd4c1a1050 con 0x7fcd4c1033f0 2024-04-28T12:48:52.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.069+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd4c0fe8d0 con 0x7fcd4c1033f0 2024-04-28T12:48:52.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.070+0000 7fcd508da640 1 -- 172.21.15.53:0/1878798344 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcd4402fd30 con 0x7fcd4c1033f0 2024-04-28T12:48:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.071+0000 7fcd508da640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcd380a3de0 0x7fcd380a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.071+0000 7fcd4a7fc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcd380a3de0 0x7fcd380a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.071+0000 7fcd508da640 1 -- 172.21.15.53:0/1878798344 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fcd440f19d0 con 0x7fcd4c1033f0 2024-04-28T12:48:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.071+0000 7fcd4a7fc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcd380a3de0 0x7fcd380a62a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcd34005fd0 tx=0x7fcd34005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.075+0000 7fcd508da640 1 -- 172.21.15.53:0/1878798344 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcd440be410 con 0x7fcd4c1033f0 2024-04-28T12:48:52.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.257+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcd4c062d70 con 0x7fcd380a3de0 2024-04-28T12:48:52.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.265+0000 7fcd508da640 1 -- 172.21.15.53:0/1878798344 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258893 (secure 0 0 0) 0x7fcd4c062d70 con 0x7fcd380a3de0 2024-04-28T12:48:52.264 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:52.266 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-04-28T12:48:52.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcd380a3de0 msgr2=0x7fcd380a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcd380a3de0 0x7fcd380a62a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcd34005fd0 tx=0x7fcd34005850 comp rx=0 tx=0).stop 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 msgr2=0x7fcd4c198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c198c60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcd44006010 tx=0x7fcd44003ba0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 shutdown_connections 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcd380a3de0 0x7fcd380a62a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4c108650 0x7fcd4c1a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4c105d20 0x7fcd4c1991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 --2- 172.21.15.53:0/1878798344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd4c1033f0 0x7fcd4c198c60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 >> 172.21.15.53:0/1878798344 conn(0x7fcd4c0fcde0 msgr2=0x7fcd4c103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 shutdown_connections 2024-04-28T12:48:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.268+0000 7fcd518dc640 1 -- 172.21.15.53:0/1878798344 wait complete. 2024-04-28T12:48:52.275 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-28T12:48:50.450427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594744,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":128,"num_read_kb":111,"num_write":266,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":237,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376932,"kb_used_data":11068,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564892,"statfs":{"total":1151940427776,"available":1151554449408,"internally_reserved":0,"allocated":11333632,"data_stored":7047294,"data_compressed":108297,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19119,"internal_metadata":374453585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4376,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":85,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.956257"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.489054+0000","last_change":"2024-04-28T12:48:44.486823+0000","last_active":"2024-04-28T12:48:44.489054+0000","last_peered":"2024-04-28T12:48:44.489054+0000","last_clean":"2024-04-28T12:48:44.489054+0000","last_became_active":"2024-04-28T12:48:44.485159+0000","last_became_peered":"2024-04-28T12:48:44.485159+0000","last_unstale":"2024-04-28T12:48:44.489054+0000","last_undegraded":"2024-04-28T12:48:44.489054+0000","last_fullsized":"2024-04-28T12:48:44.489054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:52.757190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650715+0000","last_change":"2024-04-28T12:48:44.477801+0000","last_active":"2024-04-28T12:48:45.650715+0000","last_peered":"2024-04-28T12:48:45.650715+0000","last_clean":"2024-04-28T12:48:45.650715+0000","last_became_active":"2024-04-28T12:48:44.477448+0000","last_became_peered":"2024-04-28T12:48:44.477448+0000","last_unstale":"2024-04-28T12:48:45.650715+0000","last_undegraded":"2024-04-28T12:48:45.650715+0000","last_fullsized":"2024-04-28T12:48:45.650715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:30.350306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650667+0000","last_change":"2024-04-28T12:48:44.485638+0000","last_active":"2024-04-28T12:48:45.650667+0000","last_peered":"2024-04-28T12:48:45.650667+0000","last_clean":"2024-04-28T12:48:45.650667+0000","last_became_active":"2024-04-28T12:48:44.485247+0000","last_became_peered":"2024-04-28T12:48:44.485247+0000","last_unstale":"2024-04-28T12:48:45.650667+0000","last_undegraded":"2024-04-28T12:48:45.650667+0000","last_fullsized":"2024-04-28T12:48:45.650667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:49:35.209786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.488269+0000","last_change":"2024-04-28T12:48:44.484329+0000","last_active":"2024-04-28T12:48:44.488269+0000","last_peered":"2024-04-28T12:48:44.488269+0000","last_clean":"2024-04-28T12:48:44.488269+0000","last_became_active":"2024-04-28T12:48:44.483980+0000","last_became_peered":"2024-04-28T12:48:44.483980+0000","last_unstale":"2024-04-28T12:48:44.488269+0000","last_undegraded":"2024-04-28T12:48:44.488269+0000","last_fullsized":"2024-04-28T12:48:44.488269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:15.478872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521662+0000","last_change":"2024-04-28T12:48:44.487609+0000","last_active":"2024-04-28T12:48:45.521662+0000","last_peered":"2024-04-28T12:48:45.521662+0000","last_clean":"2024-04-28T12:48:45.521662+0000","last_became_active":"2024-04-28T12:48:44.487199+0000","last_became_peered":"2024-04-28T12:48:44.487199+0000","last_unstale":"2024-04-28T12:48:45.521662+0000","last_undegraded":"2024-04-28T12:48:45.521662+0000","last_fullsized":"2024-04-28T12:48:45.521662+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:30:03.675503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650589+0000","last_change":"2024-04-28T12:48:44.477889+0000","last_active":"2024-04-28T12:48:45.650589+0000","last_peered":"2024-04-28T12:48:45.650589+0000","last_clean":"2024-04-28T12:48:45.650589+0000","last_became_active":"2024-04-28T12:48:44.477472+0000","last_became_peered":"2024-04-28T12:48:44.477472+0000","last_unstale":"2024-04-28T12:48:45.650589+0000","last_undegraded":"2024-04-28T12:48:45.650589+0000","last_fullsized":"2024-04-28T12:48:45.650589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:35:03.684380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463303+0000","last_change":"2024-04-28T12:48:44.482889+0000","last_active":"2024-04-28T12:48:45.463303+0000","last_peered":"2024-04-28T12:48:45.463303+0000","last_clean":"2024-04-28T12:48:45.463303+0000","last_became_active":"2024-04-28T12:48:44.482569+0000","last_became_peered":"2024-04-28T12:48:44.482569+0000","last_unstale":"2024-04-28T12:48:45.463303+0000","last_undegraded":"2024-04-28T12:48:45.463303+0000","last_fullsized":"2024-04-28T12:48:45.463303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:07.858792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459488+0000","last_change":"2024-04-28T12:48:44.484118+0000","last_active":"2024-04-28T12:48:45.459488+0000","last_peered":"2024-04-28T12:48:45.459488+0000","last_clean":"2024-04-28T12:48:45.459488+0000","last_became_active":"2024-04-28T12:48:44.483577+0000","last_became_peered":"2024-04-28T12:48:44.483577+0000","last_unstale":"2024-04-28T12:48:45.459488+0000","last_undegraded":"2024-04-28T12:48:45.459488+0000","last_fullsized":"2024-04-28T12:48:45.459488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:52:24.574445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.488747+0000","last_change":"2024-04-28T12:48:44.485372+0000","last_active":"2024-04-28T12:48:44.488747+0000","last_peered":"2024-04-28T12:48:44.488747+0000","last_clean":"2024-04-28T12:48:44.488747+0000","last_became_active":"2024-04-28T12:48:44.485074+0000","last_became_peered":"2024-04-28T12:48:44.485074+0000","last_unstale":"2024-04-28T12:48:44.488747+0000","last_undegraded":"2024-04-28T12:48:44.488747+0000","last_fullsized":"2024-04-28T12:48:44.488747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:20:56.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651506+0000","last_change":"2024-04-28T12:48:44.480021+0000","last_active":"2024-04-28T12:48:45.651506+0000","last_peered":"2024-04-28T12:48:45.651506+0000","last_clean":"2024-04-28T12:48:45.651506+0000","last_became_active":"2024-04-28T12:48:44.479486+0000","last_became_peered":"2024-04-28T12:48:44.479486+0000","last_unstale":"2024-04-28T12:48:45.651506+0000","last_undegraded":"2024-04-28T12:48:45.651506+0000","last_fullsized":"2024-04-28T12:48:45.651506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:04:27.885972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463843+0000","last_change":"2024-04-28T12:48:44.479273+0000","last_active":"2024-04-28T12:48:45.463843+0000","last_peered":"2024-04-28T12:48:45.463843+0000","last_clean":"2024-04-28T12:48:45.463843+0000","last_became_active":"2024-04-28T12:48:44.478962+0000","last_became_peered":"2024-04-28T12:48:44.478962+0000","last_unstale":"2024-04-28T12:48:45.463843+0000","last_undegraded":"2024-04-28T12:48:45.463843+0000","last_fullsized":"2024-04-28T12:48:45.463843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:27.698365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.481897+0000","last_change":"2024-04-28T12:48:44.478442+0000","last_active":"2024-04-28T12:48:44.481897+0000","last_peered":"2024-04-28T12:48:44.481897+0000","last_clean":"2024-04-28T12:48:44.481897+0000","last_became_active":"2024-04-28T12:48:44.478121+0000","last_became_peered":"2024-04-28T12:48:44.478121+0000","last_unstale":"2024-04-28T12:48:44.481897+0000","last_undegraded":"2024-04-28T12:48:44.481897+0000","last_fullsized":"2024-04-28T12:48:44.481897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:46:47.140476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650390+0000","last_change":"2024-04-28T12:48:44.476475+0000","last_active":"2024-04-28T12:48:45.650390+0000","last_peered":"2024-04-28T12:48:45.650390+0000","last_clean":"2024-04-28T12:48:45.650390+0000","last_became_active":"2024-04-28T12:48:44.475839+0000","last_became_peered":"2024-04-28T12:48:44.475839+0000","last_unstale":"2024-04-28T12:48:45.650390+0000","last_undegraded":"2024-04-28T12:48:45.650390+0000","last_fullsized":"2024-04-28T12:48:45.650390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:34:51.513356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650117+0000","last_change":"2024-04-28T12:48:44.480569+0000","last_active":"2024-04-28T12:48:45.650117+0000","last_peered":"2024-04-28T12:48:45.650117+0000","last_clean":"2024-04-28T12:48:45.650117+0000","last_became_active":"2024-04-28T12:48:44.478085+0000","last_became_peered":"2024-04-28T12:48:44.478085+0000","last_unstale":"2024-04-28T12:48:45.650117+0000","last_undegraded":"2024-04-28T12:48:45.650117+0000","last_fullsized":"2024-04-28T12:48:45.650117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:23:13.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.649929+0000","last_change":"2024-04-28T12:48:44.477406+0000","last_active":"2024-04-28T12:48:45.649929+0000","last_peered":"2024-04-28T12:48:45.649929+0000","last_clean":"2024-04-28T12:48:45.649929+0000","last_became_active":"2024-04-28T12:48:44.477053+0000","last_became_peered":"2024-04-28T12:48:44.477053+0000","last_unstale":"2024-04-28T12:48:45.649929+0000","last_undegraded":"2024-04-28T12:48:45.649929+0000","last_fullsized":"2024-04-28T12:48:45.649929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:29.494688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.488985+0000","last_change":"2024-04-28T12:48:44.485600+0000","last_active":"2024-04-28T12:48:44.488985+0000","last_peered":"2024-04-28T12:48:44.488985+0000","last_clean":"2024-04-28T12:48:44.488985+0000","last_became_active":"2024-04-28T12:48:44.485297+0000","last_became_peered":"2024-04-28T12:48:44.485297+0000","last_unstale":"2024-04-28T12:48:44.488985+0000","last_undegraded":"2024-04-28T12:48:44.488985+0000","last_fullsized":"2024-04-28T12:48:44.488985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:10:59.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.489075+0000","last_change":"2024-04-28T12:48:44.487376+0000","last_active":"2024-04-28T12:48:44.489075+0000","last_peered":"2024-04-28T12:48:44.489075+0000","last_clean":"2024-04-28T12:48:44.489075+0000","last_became_active":"2024-04-28T12:48:44.486561+0000","last_became_peered":"2024-04-28T12:48:44.486561+0000","last_unstale":"2024-04-28T12:48:44.489075+0000","last_undegraded":"2024-04-28T12:48:44.489075+0000","last_fullsized":"2024-04-28T12:48:44.489075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:36:04.319817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.519631+0000","last_change":"2024-04-28T12:48:44.475470+0000","last_active":"2024-04-28T12:48:45.519631+0000","last_peered":"2024-04-28T12:48:45.519631+0000","last_clean":"2024-04-28T12:48:45.519631+0000","last_became_active":"2024-04-28T12:48:44.475113+0000","last_became_peered":"2024-04-28T12:48:44.475113+0000","last_unstale":"2024-04-28T12:48:45.519631+0000","last_undegraded":"2024-04-28T12:48:45.519631+0000","last_fullsized":"2024-04-28T12:48:45.519631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:55:28.688458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650412+0000","last_change":"2024-04-28T12:48:44.482195+0000","last_active":"2024-04-28T12:48:45.650412+0000","last_peered":"2024-04-28T12:48:45.650412+0000","last_clean":"2024-04-28T12:48:45.650412+0000","last_became_active":"2024-04-28T12:48:44.481847+0000","last_became_peered":"2024-04-28T12:48:44.481847+0000","last_unstale":"2024-04-28T12:48:45.650412+0000","last_undegraded":"2024-04-28T12:48:45.650412+0000","last_fullsized":"2024-04-28T12:48:45.650412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:18:03.507012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.553256+0000","last_change":"2024-04-28T12:48:43.517808+0000","last_active":"2024-04-28T12:48:48.553256+0000","last_peered":"2024-04-28T12:48:48.553256+0000","last_clean":"2024-04-28T12:48:48.553256+0000","last_became_active":"2024-04-28T12:48:43.517443+0000","last_became_peered":"2024-04-28T12:48:43.517443+0000","last_unstale":"2024-04-28T12:48:48.553256+0000","last_undegraded":"2024-04-28T12:48:48.553256+0000","last_fullsized":"2024-04-28T12:48:48.553256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:57:11.972794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459687+0000","last_change":"2024-04-28T12:48:44.486296+0000","last_active":"2024-04-28T12:48:45.459687+0000","last_peered":"2024-04-28T12:48:45.459687+0000","last_clean":"2024-04-28T12:48:45.459687+0000","last_became_active":"2024-04-28T12:48:44.485289+0000","last_became_peered":"2024-04-28T12:48:44.485289+0000","last_unstale":"2024-04-28T12:48:45.459687+0000","last_undegraded":"2024-04-28T12:48:45.459687+0000","last_fullsized":"2024-04-28T12:48:45.459687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:07.558478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460213+0000","last_change":"2024-04-28T12:48:43.523725+0000","last_active":"2024-04-28T12:48:45.460213+0000","last_peered":"2024-04-28T12:48:45.460213+0000","last_clean":"2024-04-28T12:48:45.460213+0000","last_became_active":"2024-04-28T12:48:43.523367+0000","last_became_peered":"2024-04-28T12:48:43.523367+0000","last_unstale":"2024-04-28T12:48:45.460213+0000","last_undegraded":"2024-04-28T12:48:45.460213+0000","last_fullsized":"2024-04-28T12:48:45.460213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:15:15.674552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650922+0000","last_change":"2024-04-28T12:48:44.473393+0000","last_active":"2024-04-28T12:48:45.650922+0000","last_peered":"2024-04-28T12:48:45.650922+0000","last_clean":"2024-04-28T12:48:45.650922+0000","last_became_active":"2024-04-28T12:48:44.473035+0000","last_became_peered":"2024-04-28T12:48:44.473035+0000","last_unstale":"2024-04-28T12:48:45.650922+0000","last_undegraded":"2024-04-28T12:48:45.650922+0000","last_fullsized":"2024-04-28T12:48:45.650922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:06.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.522087+0000","last_change":"2024-04-28T12:48:43.510619+0000","last_active":"2024-04-28T12:48:45.522087+0000","last_peered":"2024-04-28T12:48:45.522087+0000","last_clean":"2024-04-28T12:48:45.522087+0000","last_became_active":"2024-04-28T12:48:43.510265+0000","last_became_peered":"2024-04-28T12:48:43.510265+0000","last_unstale":"2024-04-28T12:48:45.522087+0000","last_undegraded":"2024-04-28T12:48:45.522087+0000","last_fullsized":"2024-04-28T12:48:45.522087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:39:41.833527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.477368+0000","last_change":"2024-04-28T12:48:44.475616+0000","last_active":"2024-04-28T12:48:44.477368+0000","last_peered":"2024-04-28T12:48:44.477368+0000","last_clean":"2024-04-28T12:48:44.477368+0000","last_became_active":"2024-04-28T12:48:44.475268+0000","last_became_peered":"2024-04-28T12:48:44.475268+0000","last_unstale":"2024-04-28T12:48:44.477368+0000","last_undegraded":"2024-04-28T12:48:44.477368+0000","last_fullsized":"2024-04-28T12:48:44.477368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:18:28.174368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.522012+0000","last_change":"2024-04-28T12:48:43.517652+0000","last_active":"2024-04-28T12:48:45.522012+0000","last_peered":"2024-04-28T12:48:45.522012+0000","last_clean":"2024-04-28T12:48:45.522012+0000","last_became_active":"2024-04-28T12:48:43.517220+0000","last_became_peered":"2024-04-28T12:48:43.517220+0000","last_unstale":"2024-04-28T12:48:45.522012+0000","last_undegraded":"2024-04-28T12:48:45.522012+0000","last_fullsized":"2024-04-28T12:48:45.522012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:25:52.793542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.461224+0000","last_change":"2024-04-28T12:48:44.488860+0000","last_active":"2024-04-28T12:48:45.461224+0000","last_peered":"2024-04-28T12:48:45.461224+0000","last_clean":"2024-04-28T12:48:45.461224+0000","last_became_active":"2024-04-28T12:48:44.488487+0000","last_became_peered":"2024-04-28T12:48:44.488487+0000","last_unstale":"2024-04-28T12:48:45.461224+0000","last_undegraded":"2024-04-28T12:48:45.461224+0000","last_fullsized":"2024-04-28T12:48:45.461224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:33:28.357280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543441+0000","last_change":"2024-04-28T12:48:43.507961+0000","last_active":"2024-04-28T12:48:47.543441+0000","last_peered":"2024-04-28T12:48:47.543441+0000","last_clean":"2024-04-28T12:48:47.543441+0000","last_became_active":"2024-04-28T12:48:43.507600+0000","last_became_peered":"2024-04-28T12:48:43.507600+0000","last_unstale":"2024-04-28T12:48:47.543441+0000","last_undegraded":"2024-04-28T12:48:47.543441+0000","last_fullsized":"2024-04-28T12:48:47.543441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:16.188154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521639+0000","last_change":"2024-04-28T12:48:44.474027+0000","last_active":"2024-04-28T12:48:45.521639+0000","last_peered":"2024-04-28T12:48:45.521639+0000","last_clean":"2024-04-28T12:48:45.521639+0000","last_became_active":"2024-04-28T12:48:44.473372+0000","last_became_peered":"2024-04-28T12:48:44.473372+0000","last_unstale":"2024-04-28T12:48:45.521639+0000","last_undegraded":"2024-04-28T12:48:45.521639+0000","last_fullsized":"2024-04-28T12:48:45.521639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:18:43.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.550171+0000","last_change":"2024-04-28T12:48:43.521135+0000","last_active":"2024-04-28T12:48:48.550171+0000","last_peered":"2024-04-28T12:48:48.550171+0000","last_clean":"2024-04-28T12:48:48.550171+0000","last_became_active":"2024-04-28T12:48:43.520749+0000","last_became_peered":"2024-04-28T12:48:43.520749+0000","last_unstale":"2024-04-28T12:48:48.550171+0000","last_undegraded":"2024-04-28T12:48:48.550171+0000","last_fullsized":"2024-04-28T12:48:48.550171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:35:33.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460966+0000","last_change":"2024-04-28T12:48:44.479090+0000","last_active":"2024-04-28T12:48:45.460966+0000","last_peered":"2024-04-28T12:48:45.460966+0000","last_clean":"2024-04-28T12:48:45.460966+0000","last_became_active":"2024-04-28T12:48:44.478725+0000","last_became_peered":"2024-04-28T12:48:44.478725+0000","last_unstale":"2024-04-28T12:48:45.460966+0000","last_undegraded":"2024-04-28T12:48:45.460966+0000","last_fullsized":"2024-04-28T12:48:45.460966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:40:21.710990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.649930+0000","last_change":"2024-04-28T12:48:43.505309+0000","last_active":"2024-04-28T12:48:45.649930+0000","last_peered":"2024-04-28T12:48:45.649930+0000","last_clean":"2024-04-28T12:48:45.649930+0000","last_became_active":"2024-04-28T12:48:43.504934+0000","last_became_peered":"2024-04-28T12:48:43.504934+0000","last_unstale":"2024-04-28T12:48:45.649930+0000","last_undegraded":"2024-04-28T12:48:45.649930+0000","last_fullsized":"2024-04-28T12:48:45.649930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:48:39.652606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459557+0000","last_change":"2024-04-28T12:48:44.479930+0000","last_active":"2024-04-28T12:48:45.459557+0000","last_peered":"2024-04-28T12:48:45.459557+0000","last_clean":"2024-04-28T12:48:45.459557+0000","last_became_active":"2024-04-28T12:48:44.479571+0000","last_became_peered":"2024-04-28T12:48:44.479571+0000","last_unstale":"2024-04-28T12:48:45.459557+0000","last_undegraded":"2024-04-28T12:48:45.459557+0000","last_fullsized":"2024-04-28T12:48:45.459557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:00.130664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460641+0000","last_change":"2024-04-28T12:48:43.508935+0000","last_active":"2024-04-28T12:48:45.460641+0000","last_peered":"2024-04-28T12:48:45.460641+0000","last_clean":"2024-04-28T12:48:45.460641+0000","last_became_active":"2024-04-28T12:48:43.508596+0000","last_became_peered":"2024-04-28T12:48:43.508596+0000","last_unstale":"2024-04-28T12:48:45.460641+0000","last_undegraded":"2024-04-28T12:48:45.460641+0000","last_fullsized":"2024-04-28T12:48:45.460641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:41.905528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.519419+0000","last_change":"2024-04-28T12:48:44.489695+0000","last_active":"2024-04-28T12:48:45.519419+0000","last_peered":"2024-04-28T12:48:45.519419+0000","last_clean":"2024-04-28T12:48:45.519419+0000","last_became_active":"2024-04-28T12:48:44.489290+0000","last_became_peered":"2024-04-28T12:48:44.489290+0000","last_unstale":"2024-04-28T12:48:45.519419+0000","last_undegraded":"2024-04-28T12:48:45.519419+0000","last_fullsized":"2024-04-28T12:48:45.519419+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:34:01.873775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543827+0000","last_change":"2024-04-28T12:48:43.511751+0000","last_active":"2024-04-28T12:48:47.543827+0000","last_peered":"2024-04-28T12:48:47.543827+0000","last_clean":"2024-04-28T12:48:47.543827+0000","last_became_active":"2024-04-28T12:48:43.511428+0000","last_became_peered":"2024-04-28T12:48:43.511428+0000","last_unstale":"2024-04-28T12:48:47.543827+0000","last_undegraded":"2024-04-28T12:48:47.543827+0000","last_fullsized":"2024-04-28T12:48:47.543827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:12.987697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.477130+0000","last_change":"2024-04-28T12:48:44.472310+0000","last_active":"2024-04-28T12:48:44.477130+0000","last_peered":"2024-04-28T12:48:44.477130+0000","last_clean":"2024-04-28T12:48:44.477130+0000","last_became_active":"2024-04-28T12:48:44.471978+0000","last_became_peered":"2024-04-28T12:48:44.471978+0000","last_unstale":"2024-04-28T12:48:44.477130+0000","last_undegraded":"2024-04-28T12:48:44.477130+0000","last_fullsized":"2024-04-28T12:48:44.477130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:21:58.285778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463567+0000","last_change":"2024-04-28T12:48:43.514475+0000","last_active":"2024-04-28T12:48:45.463567+0000","last_peered":"2024-04-28T12:48:45.463567+0000","last_clean":"2024-04-28T12:48:45.463567+0000","last_became_active":"2024-04-28T12:48:43.514067+0000","last_became_peered":"2024-04-28T12:48:43.514067+0000","last_unstale":"2024-04-28T12:48:45.463567+0000","last_undegraded":"2024-04-28T12:48:45.463567+0000","last_fullsized":"2024-04-28T12:48:45.463567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:52.125456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650790+0000","last_change":"2024-04-28T12:48:44.479210+0000","last_active":"2024-04-28T12:48:45.650790+0000","last_peered":"2024-04-28T12:48:45.650790+0000","last_clean":"2024-04-28T12:48:45.650790+0000","last_became_active":"2024-04-28T12:48:44.478816+0000","last_became_peered":"2024-04-28T12:48:44.478816+0000","last_unstale":"2024-04-28T12:48:45.650790+0000","last_undegraded":"2024-04-28T12:48:45.650790+0000","last_fullsized":"2024-04-28T12:48:45.650790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:03:06.994063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.461165+0000","last_change":"2024-04-28T12:48:43.520207+0000","last_active":"2024-04-28T12:48:45.461165+0000","last_peered":"2024-04-28T12:48:45.461165+0000","last_clean":"2024-04-28T12:48:45.461165+0000","last_became_active":"2024-04-28T12:48:43.519823+0000","last_became_peered":"2024-04-28T12:48:43.519823+0000","last_unstale":"2024-04-28T12:48:45.461165+0000","last_undegraded":"2024-04-28T12:48:45.461165+0000","last_fullsized":"2024-04-28T12:48:45.461165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:02.238166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.483862+0000","last_change":"2024-04-28T12:48:44.479632+0000","last_active":"2024-04-28T12:48:44.483862+0000","last_peered":"2024-04-28T12:48:44.483862+0000","last_clean":"2024-04-28T12:48:44.483862+0000","last_became_active":"2024-04-28T12:48:44.479197+0000","last_became_peered":"2024-04-28T12:48:44.479197+0000","last_unstale":"2024-04-28T12:48:44.483862+0000","last_undegraded":"2024-04-28T12:48:44.483862+0000","last_fullsized":"2024-04-28T12:48:44.483862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:36:22.514411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543994+0000","last_change":"2024-04-28T12:48:43.514198+0000","last_active":"2024-04-28T12:48:47.543994+0000","last_peered":"2024-04-28T12:48:47.543994+0000","last_clean":"2024-04-28T12:48:47.543994+0000","last_became_active":"2024-04-28T12:48:43.513839+0000","last_became_peered":"2024-04-28T12:48:43.513839+0000","last_unstale":"2024-04-28T12:48:47.543994+0000","last_undegraded":"2024-04-28T12:48:47.543994+0000","last_fullsized":"2024-04-28T12:48:47.543994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:54:19.446610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521856+0000","last_change":"2024-04-28T12:48:44.484996+0000","last_active":"2024-04-28T12:48:45.521856+0000","last_peered":"2024-04-28T12:48:45.521856+0000","last_clean":"2024-04-28T12:48:45.521856+0000","last_became_active":"2024-04-28T12:48:44.484591+0000","last_became_peered":"2024-04-28T12:48:44.484591+0000","last_unstale":"2024-04-28T12:48:45.521856+0000","last_undegraded":"2024-04-28T12:48:45.521856+0000","last_fullsized":"2024-04-28T12:48:45.521856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:38:42.768757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.552644+0000","last_change":"2024-04-28T12:48:43.510365+0000","last_active":"2024-04-28T12:48:48.552644+0000","last_peered":"2024-04-28T12:48:48.552644+0000","last_clean":"2024-04-28T12:48:48.552644+0000","last_became_active":"2024-04-28T12:48:43.510028+0000","last_became_peered":"2024-04-28T12:48:43.510028+0000","last_unstale":"2024-04-28T12:48:48.552644+0000","last_undegraded":"2024-04-28T12:48:48.552644+0000","last_fullsized":"2024-04-28T12:48:48.552644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:09:20.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650200+0000","last_change":"2024-04-28T12:48:44.489704+0000","last_active":"2024-04-28T12:48:45.650200+0000","last_peered":"2024-04-28T12:48:45.650200+0000","last_clean":"2024-04-28T12:48:45.650200+0000","last_became_active":"2024-04-28T12:48:44.489331+0000","last_became_peered":"2024-04-28T12:48:44.489331+0000","last_unstale":"2024-04-28T12:48:45.650200+0000","last_undegraded":"2024-04-28T12:48:45.650200+0000","last_fullsized":"2024-04-28T12:48:45.650200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:33.755818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.542937+0000","last_change":"2024-04-28T12:48:43.519213+0000","last_active":"2024-04-28T12:48:47.542937+0000","last_peered":"2024-04-28T12:48:47.542937+0000","last_clean":"2024-04-28T12:48:47.542937+0000","last_became_active":"2024-04-28T12:48:43.518385+0000","last_became_peered":"2024-04-28T12:48:43.518385+0000","last_unstale":"2024-04-28T12:48:47.542937+0000","last_undegraded":"2024-04-28T12:48:47.542937+0000","last_fullsized":"2024-04-28T12:48:47.542937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:31:37.130504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463742+0000","last_change":"2024-04-28T12:48:44.480838+0000","last_active":"2024-04-28T12:48:45.463742+0000","last_peered":"2024-04-28T12:48:45.463742+0000","last_clean":"2024-04-28T12:48:45.463742+0000","last_became_active":"2024-04-28T12:48:44.480522+0000","last_became_peered":"2024-04-28T12:48:44.480522+0000","last_unstale":"2024-04-28T12:48:45.463742+0000","last_undegraded":"2024-04-28T12:48:45.463742+0000","last_fullsized":"2024-04-28T12:48:45.463742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:24.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.468097+0000","last_change":"2024-04-28T12:48:43.505498+0000","last_active":"2024-04-28T12:48:44.468097+0000","last_peered":"2024-04-28T12:48:44.468097+0000","last_clean":"2024-04-28T12:48:44.468097+0000","last_became_active":"2024-04-28T12:48:43.505041+0000","last_became_peered":"2024-04-28T12:48:43.505041+0000","last_unstale":"2024-04-28T12:48:44.468097+0000","last_undegraded":"2024-04-28T12:48:44.468097+0000","last_fullsized":"2024-04-28T12:48:44.468097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:36.624508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463192+0000","last_change":"2024-04-28T12:48:44.482698+0000","last_active":"2024-04-28T12:48:45.463192+0000","last_peered":"2024-04-28T12:48:45.463192+0000","last_clean":"2024-04-28T12:48:45.463192+0000","last_became_active":"2024-04-28T12:48:44.482365+0000","last_became_peered":"2024-04-28T12:48:44.482365+0000","last_unstale":"2024-04-28T12:48:45.463192+0000","last_undegraded":"2024-04-28T12:48:45.463192+0000","last_fullsized":"2024-04-28T12:48:45.463192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:45:33.648105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521490+0000","last_change":"2024-04-28T12:48:43.515942+0000","last_active":"2024-04-28T12:48:45.521490+0000","last_peered":"2024-04-28T12:48:45.521490+0000","last_clean":"2024-04-28T12:48:45.521490+0000","last_became_active":"2024-04-28T12:48:43.515497+0000","last_became_peered":"2024-04-28T12:48:43.515497+0000","last_unstale":"2024-04-28T12:48:45.521490+0000","last_undegraded":"2024-04-28T12:48:45.521490+0000","last_fullsized":"2024-04-28T12:48:45.521490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:39.489486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460566+0000","last_change":"2024-04-28T12:48:44.487831+0000","last_active":"2024-04-28T12:48:45.460566+0000","last_peered":"2024-04-28T12:48:45.460566+0000","last_clean":"2024-04-28T12:48:45.460566+0000","last_became_active":"2024-04-28T12:48:44.487341+0000","last_became_peered":"2024-04-28T12:48:44.487341+0000","last_unstale":"2024-04-28T12:48:45.460566+0000","last_undegraded":"2024-04-28T12:48:45.460566+0000","last_fullsized":"2024-04-28T12:48:45.460566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:42.810514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521995+0000","last_change":"2024-04-28T12:48:43.513667+0000","last_active":"2024-04-28T12:48:45.521995+0000","last_peered":"2024-04-28T12:48:45.521995+0000","last_clean":"2024-04-28T12:48:45.521995+0000","last_became_active":"2024-04-28T12:48:43.513192+0000","last_became_peered":"2024-04-28T12:48:43.513192+0000","last_unstale":"2024-04-28T12:48:45.521995+0000","last_undegraded":"2024-04-28T12:48:45.521995+0000","last_fullsized":"2024-04-28T12:48:45.521995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.470320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.480242+0000","last_change":"2024-04-28T12:48:44.478203+0000","last_active":"2024-04-28T12:48:44.480242+0000","last_peered":"2024-04-28T12:48:44.480242+0000","last_clean":"2024-04-28T12:48:44.480242+0000","last_became_active":"2024-04-28T12:48:44.476238+0000","last_became_peered":"2024-04-28T12:48:44.476238+0000","last_unstale":"2024-04-28T12:48:44.480242+0000","last_undegraded":"2024-04-28T12:48:44.480242+0000","last_fullsized":"2024-04-28T12:48:44.480242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:30:00.158827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.519327+0000","last_change":"2024-04-28T12:48:43.516036+0000","last_active":"2024-04-28T12:48:45.519327+0000","last_peered":"2024-04-28T12:48:45.519327+0000","last_clean":"2024-04-28T12:48:45.519327+0000","last_became_active":"2024-04-28T12:48:43.515494+0000","last_became_peered":"2024-04-28T12:48:43.515494+0000","last_unstale":"2024-04-28T12:48:45.519327+0000","last_undegraded":"2024-04-28T12:48:45.519327+0000","last_fullsized":"2024-04-28T12:48:45.519327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:34:56.568179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651243+0000","last_change":"2024-04-28T12:48:44.476836+0000","last_active":"2024-04-28T12:48:45.651243+0000","last_peered":"2024-04-28T12:48:45.651243+0000","last_clean":"2024-04-28T12:48:45.651243+0000","last_became_active":"2024-04-28T12:48:44.476214+0000","last_became_peered":"2024-04-28T12:48:44.476214+0000","last_unstale":"2024-04-28T12:48:45.651243+0000","last_undegraded":"2024-04-28T12:48:45.651243+0000","last_fullsized":"2024-04-28T12:48:45.651243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:24:26.707985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463308+0000","last_change":"2024-04-28T12:48:43.520918+0000","last_active":"2024-04-28T12:48:45.463308+0000","last_peered":"2024-04-28T12:48:45.463308+0000","last_clean":"2024-04-28T12:48:45.463308+0000","last_became_active":"2024-04-28T12:48:43.520564+0000","last_became_peered":"2024-04-28T12:48:43.520564+0000","last_unstale":"2024-04-28T12:48:45.463308+0000","last_undegraded":"2024-04-28T12:48:45.463308+0000","last_fullsized":"2024-04-28T12:48:45.463308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:32:51.206102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650906+0000","last_change":"2024-04-28T12:48:44.484570+0000","last_active":"2024-04-28T12:48:45.650906+0000","last_peered":"2024-04-28T12:48:45.650906+0000","last_clean":"2024-04-28T12:48:45.650906+0000","last_became_active":"2024-04-28T12:48:44.484133+0000","last_became_peered":"2024-04-28T12:48:44.484133+0000","last_unstale":"2024-04-28T12:48:45.650906+0000","last_undegraded":"2024-04-28T12:48:45.650906+0000","last_fullsized":"2024-04-28T12:48:45.650906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:59:14.274873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460829+0000","last_change":"2024-04-28T12:48:44.478013+0000","last_active":"2024-04-28T12:48:45.460829+0000","last_peered":"2024-04-28T12:48:45.460829+0000","last_clean":"2024-04-28T12:48:45.460829+0000","last_became_active":"2024-04-28T12:48:44.477559+0000","last_became_peered":"2024-04-28T12:48:44.477559+0000","last_unstale":"2024-04-28T12:48:45.460829+0000","last_undegraded":"2024-04-28T12:48:45.460829+0000","last_fullsized":"2024-04-28T12:48:45.460829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:57:49.931865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.649776+0000","last_change":"2024-04-28T12:48:43.515873+0000","last_active":"2024-04-28T12:48:45.649776+0000","last_peered":"2024-04-28T12:48:45.649776+0000","last_clean":"2024-04-28T12:48:45.649776+0000","last_became_active":"2024-04-28T12:48:43.515442+0000","last_became_peered":"2024-04-28T12:48:43.515442+0000","last_unstale":"2024-04-28T12:48:45.649776+0000","last_undegraded":"2024-04-28T12:48:45.649776+0000","last_fullsized":"2024-04-28T12:48:45.649776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:29:57.973433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459795+0000","last_change":"2024-04-28T12:48:44.479868+0000","last_active":"2024-04-28T12:48:45.459795+0000","last_peered":"2024-04-28T12:48:45.459795+0000","last_clean":"2024-04-28T12:48:45.459795+0000","last_became_active":"2024-04-28T12:48:44.479519+0000","last_became_peered":"2024-04-28T12:48:44.479519+0000","last_unstale":"2024-04-28T12:48:45.459795+0000","last_undegraded":"2024-04-28T12:48:45.459795+0000","last_fullsized":"2024-04-28T12:48:45.459795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:09:20.946380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.465140+0000","last_change":"2024-04-28T12:48:43.507662+0000","last_active":"2024-04-28T12:48:44.465140+0000","last_peered":"2024-04-28T12:48:44.465140+0000","last_clean":"2024-04-28T12:48:44.465140+0000","last_became_active":"2024-04-28T12:48:43.507331+0000","last_became_peered":"2024-04-28T12:48:43.507331+0000","last_unstale":"2024-04-28T12:48:44.465140+0000","last_undegraded":"2024-04-28T12:48:44.465140+0000","last_fullsized":"2024-04-28T12:48:44.465140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:10.073569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":338,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463187+0000","last_change":"2024-04-28T12:47:27.444407+0000","last_active":"2024-04-28T12:48:45.463187+0000","last_peered":"2024-04-28T12:48:45.463187+0000","last_clean":"2024-04-28T12:48:45.463187+0000","last_became_active":"2024-04-28T12:47:27.443992+0000","last_became_peered":"2024-04-28T12:47:27.443992+0000","last_unstale":"2024-04-28T12:48:45.463187+0000","last_undegraded":"2024-04-28T12:48:45.463187+0000","last_fullsized":"2024-04-28T12:48:45.463187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_clean_scrub_stamp":"2024-04-28T12:44:00.636011+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:17:23.614524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.649824+0000","last_change":"2024-04-28T12:48:44.479395+0000","last_active":"2024-04-28T12:48:45.649824+0000","last_peered":"2024-04-28T12:48:45.649824+0000","last_clean":"2024-04-28T12:48:45.649824+0000","last_became_active":"2024-04-28T12:48:44.478935+0000","last_became_peered":"2024-04-28T12:48:44.478935+0000","last_unstale":"2024-04-28T12:48:45.649824+0000","last_undegraded":"2024-04-28T12:48:45.649824+0000","last_fullsized":"2024-04-28T12:48:45.649824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:19:28.212744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463431+0000","last_change":"2024-04-28T12:48:43.512026+0000","last_active":"2024-04-28T12:48:45.463431+0000","last_peered":"2024-04-28T12:48:45.463431+0000","last_clean":"2024-04-28T12:48:45.463431+0000","last_became_active":"2024-04-28T12:48:43.511650+0000","last_became_peered":"2024-04-28T12:48:43.511650+0000","last_unstale":"2024-04-28T12:48:45.463431+0000","last_undegraded":"2024-04-28T12:48:45.463431+0000","last_fullsized":"2024-04-28T12:48:45.463431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:38:04.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.522010+0000","last_change":"2024-04-28T12:48:43.520068+0000","last_active":"2024-04-28T12:48:45.522010+0000","last_peered":"2024-04-28T12:48:45.522010+0000","last_clean":"2024-04-28T12:48:45.522010+0000","last_became_active":"2024-04-28T12:48:43.519624+0000","last_became_peered":"2024-04-28T12:48:43.519624+0000","last_unstale":"2024-04-28T12:48:45.522010+0000","last_undegraded":"2024-04-28T12:48:45.522010+0000","last_fullsized":"2024-04-28T12:48:45.522010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:35:45.942437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650390+0000","last_change":"2024-04-28T12:48:44.470458+0000","last_active":"2024-04-28T12:48:45.650390+0000","last_peered":"2024-04-28T12:48:45.650390+0000","last_clean":"2024-04-28T12:48:45.650390+0000","last_became_active":"2024-04-28T12:48:44.470105+0000","last_became_peered":"2024-04-28T12:48:44.470105+0000","last_unstale":"2024-04-28T12:48:45.650390+0000","last_undegraded":"2024-04-28T12:48:45.650390+0000","last_fullsized":"2024-04-28T12:48:45.650390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:49:25.970990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.551971+0000","last_change":"2024-04-28T12:48:43.510065+0000","last_active":"2024-04-28T12:48:48.551971+0000","last_peered":"2024-04-28T12:48:48.551971+0000","last_clean":"2024-04-28T12:48:48.551971+0000","last_became_active":"2024-04-28T12:48:43.509718+0000","last_became_peered":"2024-04-28T12:48:43.509718+0000","last_unstale":"2024-04-28T12:48:48.551971+0000","last_undegraded":"2024-04-28T12:48:48.551971+0000","last_fullsized":"2024-04-28T12:48:48.551971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:50:46.776100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459879+0000","last_change":"2024-04-28T12:48:44.487022+0000","last_active":"2024-04-28T12:48:45.459879+0000","last_peered":"2024-04-28T12:48:45.459879+0000","last_clean":"2024-04-28T12:48:45.459879+0000","last_became_active":"2024-04-28T12:48:44.486675+0000","last_became_peered":"2024-04-28T12:48:44.486675+0000","last_unstale":"2024-04-28T12:48:45.459879+0000","last_undegraded":"2024-04-28T12:48:45.459879+0000","last_fullsized":"2024-04-28T12:48:45.459879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:36.185845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.524167+0000","last_change":"2024-04-28T12:48:43.505464+0000","last_active":"2024-04-28T12:48:44.524167+0000","last_peered":"2024-04-28T12:48:44.524167+0000","last_clean":"2024-04-28T12:48:44.524167+0000","last_became_active":"2024-04-28T12:48:43.505085+0000","last_became_peered":"2024-04-28T12:48:43.505085+0000","last_unstale":"2024-04-28T12:48:44.524167+0000","last_undegraded":"2024-04-28T12:48:44.524167+0000","last_fullsized":"2024-04-28T12:48:44.524167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:37:47.875243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650740+0000","last_change":"2024-04-28T12:48:44.486517+0000","last_active":"2024-04-28T12:48:45.650740+0000","last_peered":"2024-04-28T12:48:45.650740+0000","last_clean":"2024-04-28T12:48:45.650740+0000","last_became_active":"2024-04-28T12:48:44.486139+0000","last_became_peered":"2024-04-28T12:48:44.486139+0000","last_unstale":"2024-04-28T12:48:45.650740+0000","last_undegraded":"2024-04-28T12:48:45.650740+0000","last_fullsized":"2024-04-28T12:48:45.650740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:32:23.762496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.465871+0000","last_change":"2024-04-28T12:48:43.508992+0000","last_active":"2024-04-28T12:48:44.465871+0000","last_peered":"2024-04-28T12:48:44.465871+0000","last_clean":"2024-04-28T12:48:44.465871+0000","last_became_active":"2024-04-28T12:48:43.508492+0000","last_became_peered":"2024-04-28T12:48:43.508492+0000","last_unstale":"2024-04-28T12:48:44.465871+0000","last_undegraded":"2024-04-28T12:48:44.465871+0000","last_fullsized":"2024-04-28T12:48:44.465871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:56:29.138232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521559+0000","last_change":"2024-04-28T12:48:44.476462+0000","last_active":"2024-04-28T12:48:45.521559+0000","last_peered":"2024-04-28T12:48:45.521559+0000","last_clean":"2024-04-28T12:48:45.521559+0000","last_became_active":"2024-04-28T12:48:44.475660+0000","last_became_peered":"2024-04-28T12:48:44.475660+0000","last_unstale":"2024-04-28T12:48:45.521559+0000","last_undegraded":"2024-04-28T12:48:45.521559+0000","last_fullsized":"2024-04-28T12:48:45.521559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:49:48.656837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.542424+0000","last_change":"2024-04-28T12:48:43.508946+0000","last_active":"2024-04-28T12:48:47.542424+0000","last_peered":"2024-04-28T12:48:47.542424+0000","last_clean":"2024-04-28T12:48:47.542424+0000","last_became_active":"2024-04-28T12:48:43.508553+0000","last_became_peered":"2024-04-28T12:48:43.508553+0000","last_unstale":"2024-04-28T12:48:47.542424+0000","last_undegraded":"2024-04-28T12:48:47.542424+0000","last_fullsized":"2024-04-28T12:48:47.542424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:52:56.461509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.474843+0000","last_change":"2024-04-28T12:48:44.473967+0000","last_active":"2024-04-28T12:48:44.474843+0000","last_peered":"2024-04-28T12:48:44.474843+0000","last_clean":"2024-04-28T12:48:44.474843+0000","last_became_active":"2024-04-28T12:48:44.473614+0000","last_became_peered":"2024-04-28T12:48:44.473614+0000","last_unstale":"2024-04-28T12:48:44.474843+0000","last_undegraded":"2024-04-28T12:48:44.474843+0000","last_fullsized":"2024-04-28T12:48:44.474843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:01:15.574524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650347+0000","last_change":"2024-04-28T12:48:43.511287+0000","last_active":"2024-04-28T12:48:45.650347+0000","last_peered":"2024-04-28T12:48:45.650347+0000","last_clean":"2024-04-28T12:48:45.650347+0000","last_became_active":"2024-04-28T12:48:43.510955+0000","last_became_peered":"2024-04-28T12:48:43.510955+0000","last_unstale":"2024-04-28T12:48:45.650347+0000","last_undegraded":"2024-04-28T12:48:45.650347+0000","last_fullsized":"2024-04-28T12:48:45.650347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:40.952405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459880+0000","last_change":"2024-04-28T12:48:44.480514+0000","last_active":"2024-04-28T12:48:45.459880+0000","last_peered":"2024-04-28T12:48:45.459880+0000","last_clean":"2024-04-28T12:48:45.459880+0000","last_became_active":"2024-04-28T12:48:44.480160+0000","last_became_peered":"2024-04-28T12:48:44.480160+0000","last_unstale":"2024-04-28T12:48:45.459880+0000","last_undegraded":"2024-04-28T12:48:45.459880+0000","last_fullsized":"2024-04-28T12:48:45.459880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:07:02.221492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521632+0000","last_change":"2024-04-28T12:48:43.511823+0000","last_active":"2024-04-28T12:48:45.521632+0000","last_peered":"2024-04-28T12:48:45.521632+0000","last_clean":"2024-04-28T12:48:45.521632+0000","last_became_active":"2024-04-28T12:48:43.511420+0000","last_became_peered":"2024-04-28T12:48:43.511420+0000","last_unstale":"2024-04-28T12:48:45.521632+0000","last_undegraded":"2024-04-28T12:48:45.521632+0000","last_fullsized":"2024-04-28T12:48:45.521632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:59:10.170003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463551+0000","last_change":"2024-04-28T12:48:44.474095+0000","last_active":"2024-04-28T12:48:45.463551+0000","last_peered":"2024-04-28T12:48:45.463551+0000","last_clean":"2024-04-28T12:48:45.463551+0000","last_became_active":"2024-04-28T12:48:44.473377+0000","last_became_peered":"2024-04-28T12:48:44.473377+0000","last_unstale":"2024-04-28T12:48:45.463551+0000","last_undegraded":"2024-04-28T12:48:45.463551+0000","last_fullsized":"2024-04-28T12:48:45.463551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.355527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.552770+0000","last_change":"2024-04-28T12:48:43.515654+0000","last_active":"2024-04-28T12:48:48.552770+0000","last_peered":"2024-04-28T12:48:48.552770+0000","last_clean":"2024-04-28T12:48:48.552770+0000","last_became_active":"2024-04-28T12:48:43.515056+0000","last_became_peered":"2024-04-28T12:48:43.515056+0000","last_unstale":"2024-04-28T12:48:48.552770+0000","last_undegraded":"2024-04-28T12:48:48.552770+0000","last_fullsized":"2024-04-28T12:48:48.552770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:47:58.297248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463982+0000","last_change":"2024-04-28T12:48:44.474106+0000","last_active":"2024-04-28T12:48:45.463982+0000","last_peered":"2024-04-28T12:48:45.463982+0000","last_clean":"2024-04-28T12:48:45.463982+0000","last_became_active":"2024-04-28T12:48:44.473625+0000","last_became_peered":"2024-04-28T12:48:44.473625+0000","last_unstale":"2024-04-28T12:48:45.463982+0000","last_undegraded":"2024-04-28T12:48:45.463982+0000","last_fullsized":"2024-04-28T12:48:45.463982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:28:44.434155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460973+0000","last_change":"2024-04-28T12:48:43.519658+0000","last_active":"2024-04-28T12:48:45.460973+0000","last_peered":"2024-04-28T12:48:45.460973+0000","last_clean":"2024-04-28T12:48:45.460973+0000","last_became_active":"2024-04-28T12:48:43.519291+0000","last_became_peered":"2024-04-28T12:48:43.519291+0000","last_unstale":"2024-04-28T12:48:45.460973+0000","last_undegraded":"2024-04-28T12:48:45.460973+0000","last_fullsized":"2024-04-28T12:48:45.460973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:41.356834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463558+0000","last_change":"2024-04-28T12:48:44.483790+0000","last_active":"2024-04-28T12:48:45.463558+0000","last_peered":"2024-04-28T12:48:45.463558+0000","last_clean":"2024-04-28T12:48:45.463558+0000","last_became_active":"2024-04-28T12:48:44.483442+0000","last_became_peered":"2024-04-28T12:48:44.483442+0000","last_unstale":"2024-04-28T12:48:45.463558+0000","last_undegraded":"2024-04-28T12:48:45.463558+0000","last_fullsized":"2024-04-28T12:48:45.463558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:38:59.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650474+0000","last_change":"2024-04-28T12:48:43.505650+0000","last_active":"2024-04-28T12:48:45.650474+0000","last_peered":"2024-04-28T12:48:45.650474+0000","last_clean":"2024-04-28T12:48:45.650474+0000","last_became_active":"2024-04-28T12:48:43.505305+0000","last_became_peered":"2024-04-28T12:48:43.505305+0000","last_unstale":"2024-04-28T12:48:45.650474+0000","last_undegraded":"2024-04-28T12:48:45.650474+0000","last_fullsized":"2024-04-28T12:48:45.650474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:56:34.654105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463834+0000","last_change":"2024-04-28T12:48:44.478335+0000","last_active":"2024-04-28T12:48:45.463834+0000","last_peered":"2024-04-28T12:48:45.463834+0000","last_clean":"2024-04-28T12:48:45.463834+0000","last_became_active":"2024-04-28T12:48:44.477958+0000","last_became_peered":"2024-04-28T12:48:44.477958+0000","last_unstale":"2024-04-28T12:48:45.463834+0000","last_undegraded":"2024-04-28T12:48:45.463834+0000","last_fullsized":"2024-04-28T12:48:45.463834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:27.168011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.482488+0000","last_change":"2024-04-28T12:48:44.479601+0000","last_active":"2024-04-28T12:48:44.482488+0000","last_peered":"2024-04-28T12:48:44.482488+0000","last_clean":"2024-04-28T12:48:44.482488+0000","last_became_active":"2024-04-28T12:48:44.479262+0000","last_became_peered":"2024-04-28T12:48:44.479262+0000","last_unstale":"2024-04-28T12:48:44.482488+0000","last_undegraded":"2024-04-28T12:48:44.482488+0000","last_fullsized":"2024-04-28T12:48:44.482488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:20:24.008470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650168+0000","last_change":"2024-04-28T12:48:44.478553+0000","last_active":"2024-04-28T12:48:45.650168+0000","last_peered":"2024-04-28T12:48:45.650168+0000","last_clean":"2024-04-28T12:48:45.650168+0000","last_became_active":"2024-04-28T12:48:44.477539+0000","last_became_peered":"2024-04-28T12:48:44.477539+0000","last_unstale":"2024-04-28T12:48:45.650168+0000","last_undegraded":"2024-04-28T12:48:45.650168+0000","last_fullsized":"2024-04-28T12:48:45.650168+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:03:44.149337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650143+0000","last_change":"2024-04-28T12:48:44.483534+0000","last_active":"2024-04-28T12:48:45.650143+0000","last_peered":"2024-04-28T12:48:45.650143+0000","last_clean":"2024-04-28T12:48:45.650143+0000","last_became_active":"2024-04-28T12:48:44.483155+0000","last_became_peered":"2024-04-28T12:48:44.483155+0000","last_unstale":"2024-04-28T12:48:45.650143+0000","last_undegraded":"2024-04-28T12:48:45.650143+0000","last_fullsized":"2024-04-28T12:48:45.650143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:28:39.903443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650285+0000","last_change":"2024-04-28T12:48:44.479729+0000","last_active":"2024-04-28T12:48:45.650285+0000","last_peered":"2024-04-28T12:48:45.650285+0000","last_clean":"2024-04-28T12:48:45.650285+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:48:45.650285+0000","last_undegraded":"2024-04-28T12:48:45.650285+0000","last_fullsized":"2024-04-28T12:48:45.650285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:33:32.456549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650595+0000","last_change":"2024-04-28T12:48:44.486785+0000","last_active":"2024-04-28T12:48:45.650595+0000","last_peered":"2024-04-28T12:48:45.650595+0000","last_clean":"2024-04-28T12:48:45.650595+0000","last_became_active":"2024-04-28T12:48:44.485634+0000","last_became_peered":"2024-04-28T12:48:44.485634+0000","last_unstale":"2024-04-28T12:48:45.650595+0000","last_undegraded":"2024-04-28T12:48:45.650595+0000","last_fullsized":"2024-04-28T12:48:45.650595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:58.148545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.482520+0000","last_change":"2024-04-28T12:48:44.479690+0000","last_active":"2024-04-28T12:48:44.482520+0000","last_peered":"2024-04-28T12:48:44.482520+0000","last_clean":"2024-04-28T12:48:44.482520+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:48:44.482520+0000","last_undegraded":"2024-04-28T12:48:44.482520+0000","last_fullsized":"2024-04-28T12:48:44.482520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:28:59.916208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650565+0000","last_change":"2024-04-28T12:48:44.472305+0000","last_active":"2024-04-28T12:48:45.650565+0000","last_peered":"2024-04-28T12:48:45.650565+0000","last_clean":"2024-04-28T12:48:45.650565+0000","last_became_active":"2024-04-28T12:48:44.471227+0000","last_became_peered":"2024-04-28T12:48:44.471227+0000","last_unstale":"2024-04-28T12:48:45.650565+0000","last_undegraded":"2024-04-28T12:48:45.650565+0000","last_fullsized":"2024-04-28T12:48:45.650565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:27:14.602075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459670+0000","last_change":"2024-04-28T12:48:44.490866+0000","last_active":"2024-04-28T12:48:45.459670+0000","last_peered":"2024-04-28T12:48:45.459670+0000","last_clean":"2024-04-28T12:48:45.459670+0000","last_became_active":"2024-04-28T12:48:44.490515+0000","last_became_peered":"2024-04-28T12:48:44.490515+0000","last_unstale":"2024-04-28T12:48:45.459670+0000","last_undegraded":"2024-04-28T12:48:45.459670+0000","last_fullsized":"2024-04-28T12:48:45.459670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:19:22.404853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460659+0000","last_change":"2024-04-28T12:48:44.480611+0000","last_active":"2024-04-28T12:48:45.460659+0000","last_peered":"2024-04-28T12:48:45.460659+0000","last_clean":"2024-04-28T12:48:45.460659+0000","last_became_active":"2024-04-28T12:48:44.480294+0000","last_became_peered":"2024-04-28T12:48:44.480294+0000","last_unstale":"2024-04-28T12:48:45.460659+0000","last_undegraded":"2024-04-28T12:48:45.460659+0000","last_fullsized":"2024-04-28T12:48:45.460659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:55.450825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459582+0000","last_change":"2024-04-28T12:48:44.483003+0000","last_active":"2024-04-28T12:48:45.459582+0000","last_peered":"2024-04-28T12:48:45.459582+0000","last_clean":"2024-04-28T12:48:45.459582+0000","last_became_active":"2024-04-28T12:48:44.482630+0000","last_became_peered":"2024-04-28T12:48:44.482630+0000","last_unstale":"2024-04-28T12:48:45.459582+0000","last_undegraded":"2024-04-28T12:48:45.459582+0000","last_fullsized":"2024-04-28T12:48:45.459582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:05:58.358655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.490876+0000","last_change":"2024-04-28T12:48:44.489780+0000","last_active":"2024-04-28T12:48:44.490876+0000","last_peered":"2024-04-28T12:48:44.490876+0000","last_clean":"2024-04-28T12:48:44.490876+0000","last_became_active":"2024-04-28T12:48:44.489276+0000","last_became_peered":"2024-04-28T12:48:44.489276+0000","last_unstale":"2024-04-28T12:48:44.490876+0000","last_undegraded":"2024-04-28T12:48:44.490876+0000","last_fullsized":"2024-04-28T12:48:44.490876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:54:25.079330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650486+0000","last_change":"2024-04-28T12:48:44.472196+0000","last_active":"2024-04-28T12:48:45.650486+0000","last_peered":"2024-04-28T12:48:45.650486+0000","last_clean":"2024-04-28T12:48:45.650486+0000","last_became_active":"2024-04-28T12:48:44.471857+0000","last_became_peered":"2024-04-28T12:48:44.471857+0000","last_unstale":"2024-04-28T12:48:45.650486+0000","last_undegraded":"2024-04-28T12:48:45.650486+0000","last_fullsized":"2024-04-28T12:48:45.650486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:43.813225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-28T12:48:44.480269+0000","last_change":"2024-04-28T12:48:44.478788+0000","last_active":"2024-04-28T12:48:44.480269+0000","last_peered":"2024-04-28T12:48:44.480269+0000","last_clean":"2024-04-28T12:48:44.480269+0000","last_became_active":"2024-04-28T12:48:44.478449+0000","last_became_peered":"2024-04-28T12:48:44.478449+0000","last_unstale":"2024-04-28T12:48:44.480269+0000","last_undegraded":"2024-04-28T12:48:44.480269+0000","last_fullsized":"2024-04-28T12:48:44.480269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:25:50.562563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283884,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4376,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":85,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":11640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":909312,"data_stored":433002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":36174274},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037790,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":929792,"data_stored":442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299427,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":1032522,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1587,"internal_metadata":31914445},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":921600,"data_stored":434614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":913408,"data_stored":433092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":444534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":441856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":433024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724589,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":1033416,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:48:52.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-28T12:48:52.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 -- 172.21.15.53:0/4211936374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374101890 msgr2=0x7f7374101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 --2- 172.21.15.53:0/4211936374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374101890 0x7f7374101cb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7368009900 tx=0x7f736802a410 comp rx=0 tx=0).stop 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 -- 172.21.15.53:0/4211936374 shutdown_connections 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 --2- 172.21.15.53:0/4211936374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7374103440 0x7f7374109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 --2- 172.21.15.53:0/4211936374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7374102a80 0x7f7374102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 --2- 172.21.15.53:0/4211936374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374101890 0x7f7374101cb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 -- 172.21.15.53:0/4211936374 >> 172.21.15.53:0/4211936374 conn(0x7f73740fce20 msgr2=0x7f73740ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 -- 172.21.15.53:0/4211936374 shutdown_connections 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.421+0000 7f737a8cf640 1 -- 172.21.15.53:0/4211936374 wait complete. 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 Processor -- start 2024-04-28T12:48:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 -- start start 2024-04-28T12:48:52.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7374101890 0x7f737410c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7374102a80 0x7f737410a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374103440 0x7f737410ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f737410e4e0 con 0x7f7374103440 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f737410e360 con 0x7f7374101890 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f737a8cf640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f737410e660 con 0x7f7374102a80 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f7378e6c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374103440 0x7f737410ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f7378e6c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374103440 0x7f737410ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38696/0 (socket says 172.21.15.53:38696) 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.422+0000 7f7378e6c640 1 -- 172.21.15.53:0/3547670307 learned_addr learned my addr 172.21.15.53:0/3547670307 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f7373fff640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7374101890 0x7f737410c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f73737fe640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7374102a80 0x7f737410a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f7378e6c640 1 -- 172.21.15.53:0/3547670307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7374102a80 msgr2=0x7f737410a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f7378e6c640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7374102a80 0x7f737410a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f7378e6c640 1 -- 172.21.15.53:0/3547670307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7374101890 msgr2=0x7f737410c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f7378e6c640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7374101890 0x7f737410c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f7378e6c640 1 -- 172.21.15.53:0/3547670307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737410b620 con 0x7f7374103440 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f7378e6c640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374103440 0x7f737410ad90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f736c00eaa0 tx=0x7f736c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f73717fa640 1 -- 172.21.15.53:0/3547670307 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c00cdb0 con 0x7f7374103440 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73741a2d40 con 0x7f7374103440 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f73717fa640 1 -- 172.21.15.53:0/3547670307 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f736c005020 con 0x7f7374103440 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f73717fa640 1 -- 172.21.15.53:0/3547670307 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c0116d0 con 0x7f7374103440 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.423+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73741a3180 con 0x7f7374103440 2024-04-28T12:48:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.424+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7374108d40 con 0x7f7374103440 2024-04-28T12:48:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.425+0000 7f73717fa640 1 -- 172.21.15.53:0/3547670307 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f736c0051d0 con 0x7f7374103440 2024-04-28T12:48:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.426+0000 7f73717fa640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f735c0a3c40 0x7f735c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.426+0000 7f73717fa640 1 -- 172.21.15.53:0/3547670307 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f736c0d0930 con 0x7f7374103440 2024-04-28T12:48:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.426+0000 7f7373fff640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f735c0a3c40 0x7f735c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.427+0000 7f7373fff640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f735c0a3c40 0x7f735c0a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7368009230 tx=0x7f73680314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.430+0000 7f73717fa640 1 -- 172.21.15.53:0/3547670307 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f736c016030 con 0x7f7374103440 2024-04-28T12:48:52.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.620+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7374062d70 con 0x7f735c0a3c40 2024-04-28T12:48:52.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.628+0000 7f73717fa640 1 -- 172.21.15.53:0/3547670307 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258872 (secure 0 0 0) 0x7f7374062d70 con 0x7f735c0a3c40 2024-04-28T12:48:52.627 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:52.628 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-04-28T12:48:52.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f735c0a3c40 msgr2=0x7f735c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f735c0a3c40 0x7f735c0a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7368009230 tx=0x7f73680314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374103440 msgr2=0x7f737410ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374103440 0x7f737410ad90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f736c00eaa0 tx=0x7f736c00ef60 comp rx=0 tx=0).stop 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 shutdown_connections 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f735c0a3c40 0x7f735c0a6100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7374103440 0x7f737410ad90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7374102a80 0x7f737410a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 --2- 172.21.15.53:0/3547670307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7374101890 0x7f737410c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 >> 172.21.15.53:0/3547670307 conn(0x7f73740fce20 msgr2=0x7f7374105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 shutdown_connections 2024-04-28T12:48:52.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:52.631+0000 7f737a8cf640 1 -- 172.21.15.53:0/3547670307 wait complete. 2024-04-28T12:48:52.638 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-04-28T12:48:52.450862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595707,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":143,"num_read_kb":124,"num_write":307,"num_write_kb":2977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376992,"kb_used_data":11192,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564832,"statfs":{"total":1151940427776,"available":1151554387968,"internally_reserved":0,"allocated":11460608,"data_stored":7092434,"data_compressed":108297,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19119,"internal_metadata":374453585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5339,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":126,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002114"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.735925+0000","last_change":"2024-04-28T12:48:44.486823+0000","last_active":"2024-04-28T12:48:45.735925+0000","last_peered":"2024-04-28T12:48:45.735925+0000","last_clean":"2024-04-28T12:48:45.735925+0000","last_became_active":"2024-04-28T12:48:44.485159+0000","last_became_peered":"2024-04-28T12:48:44.485159+0000","last_unstale":"2024-04-28T12:48:45.735925+0000","last_undegraded":"2024-04-28T12:48:45.735925+0000","last_fullsized":"2024-04-28T12:48:45.735925+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:12.126908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650715+0000","last_change":"2024-04-28T12:48:44.477801+0000","last_active":"2024-04-28T12:48:45.650715+0000","last_peered":"2024-04-28T12:48:45.650715+0000","last_clean":"2024-04-28T12:48:45.650715+0000","last_became_active":"2024-04-28T12:48:44.477448+0000","last_became_peered":"2024-04-28T12:48:44.477448+0000","last_unstale":"2024-04-28T12:48:45.650715+0000","last_undegraded":"2024-04-28T12:48:45.650715+0000","last_fullsized":"2024-04-28T12:48:45.650715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:30.350306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650667+0000","last_change":"2024-04-28T12:48:44.485638+0000","last_active":"2024-04-28T12:48:45.650667+0000","last_peered":"2024-04-28T12:48:45.650667+0000","last_clean":"2024-04-28T12:48:45.650667+0000","last_became_active":"2024-04-28T12:48:44.485247+0000","last_became_peered":"2024-04-28T12:48:44.485247+0000","last_unstale":"2024-04-28T12:48:45.650667+0000","last_undegraded":"2024-04-28T12:48:45.650667+0000","last_fullsized":"2024-04-28T12:48:45.650667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:49:35.209786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651267+0000","last_change":"2024-04-28T12:48:44.484329+0000","last_active":"2024-04-28T12:48:45.651267+0000","last_peered":"2024-04-28T12:48:45.651267+0000","last_clean":"2024-04-28T12:48:45.651267+0000","last_became_active":"2024-04-28T12:48:44.483980+0000","last_became_peered":"2024-04-28T12:48:44.483980+0000","last_unstale":"2024-04-28T12:48:45.651267+0000","last_undegraded":"2024-04-28T12:48:45.651267+0000","last_fullsized":"2024-04-28T12:48:45.651267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:14:47.570749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521662+0000","last_change":"2024-04-28T12:48:44.487609+0000","last_active":"2024-04-28T12:48:45.521662+0000","last_peered":"2024-04-28T12:48:45.521662+0000","last_clean":"2024-04-28T12:48:45.521662+0000","last_became_active":"2024-04-28T12:48:44.487199+0000","last_became_peered":"2024-04-28T12:48:44.487199+0000","last_unstale":"2024-04-28T12:48:45.521662+0000","last_undegraded":"2024-04-28T12:48:45.521662+0000","last_fullsized":"2024-04-28T12:48:45.521662+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:30:03.675503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650589+0000","last_change":"2024-04-28T12:48:44.477889+0000","last_active":"2024-04-28T12:48:45.650589+0000","last_peered":"2024-04-28T12:48:45.650589+0000","last_clean":"2024-04-28T12:48:45.650589+0000","last_became_active":"2024-04-28T12:48:44.477472+0000","last_became_peered":"2024-04-28T12:48:44.477472+0000","last_unstale":"2024-04-28T12:48:45.650589+0000","last_undegraded":"2024-04-28T12:48:45.650589+0000","last_fullsized":"2024-04-28T12:48:45.650589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:35:03.684380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463303+0000","last_change":"2024-04-28T12:48:44.482889+0000","last_active":"2024-04-28T12:48:45.463303+0000","last_peered":"2024-04-28T12:48:45.463303+0000","last_clean":"2024-04-28T12:48:45.463303+0000","last_became_active":"2024-04-28T12:48:44.482569+0000","last_became_peered":"2024-04-28T12:48:44.482569+0000","last_unstale":"2024-04-28T12:48:45.463303+0000","last_undegraded":"2024-04-28T12:48:45.463303+0000","last_fullsized":"2024-04-28T12:48:45.463303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:07.858792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459488+0000","last_change":"2024-04-28T12:48:44.484118+0000","last_active":"2024-04-28T12:48:45.459488+0000","last_peered":"2024-04-28T12:48:45.459488+0000","last_clean":"2024-04-28T12:48:45.459488+0000","last_became_active":"2024-04-28T12:48:44.483577+0000","last_became_peered":"2024-04-28T12:48:44.483577+0000","last_unstale":"2024-04-28T12:48:45.459488+0000","last_undegraded":"2024-04-28T12:48:45.459488+0000","last_fullsized":"2024-04-28T12:48:45.459488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:52:24.574445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651479+0000","last_change":"2024-04-28T12:48:44.485372+0000","last_active":"2024-04-28T12:48:45.651479+0000","last_peered":"2024-04-28T12:48:45.651479+0000","last_clean":"2024-04-28T12:48:45.651479+0000","last_became_active":"2024-04-28T12:48:44.485074+0000","last_became_peered":"2024-04-28T12:48:44.485074+0000","last_unstale":"2024-04-28T12:48:45.651479+0000","last_undegraded":"2024-04-28T12:48:45.651479+0000","last_fullsized":"2024-04-28T12:48:45.651479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:24:39.390059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651506+0000","last_change":"2024-04-28T12:48:44.480021+0000","last_active":"2024-04-28T12:48:45.651506+0000","last_peered":"2024-04-28T12:48:45.651506+0000","last_clean":"2024-04-28T12:48:45.651506+0000","last_became_active":"2024-04-28T12:48:44.479486+0000","last_became_peered":"2024-04-28T12:48:44.479486+0000","last_unstale":"2024-04-28T12:48:45.651506+0000","last_undegraded":"2024-04-28T12:48:45.651506+0000","last_fullsized":"2024-04-28T12:48:45.651506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:04:27.885972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463843+0000","last_change":"2024-04-28T12:48:44.479273+0000","last_active":"2024-04-28T12:48:45.463843+0000","last_peered":"2024-04-28T12:48:45.463843+0000","last_clean":"2024-04-28T12:48:45.463843+0000","last_became_active":"2024-04-28T12:48:44.478962+0000","last_became_peered":"2024-04-28T12:48:44.478962+0000","last_unstale":"2024-04-28T12:48:45.463843+0000","last_undegraded":"2024-04-28T12:48:45.463843+0000","last_fullsized":"2024-04-28T12:48:45.463843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:27.698365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651099+0000","last_change":"2024-04-28T12:48:44.478442+0000","last_active":"2024-04-28T12:48:45.651099+0000","last_peered":"2024-04-28T12:48:45.651099+0000","last_clean":"2024-04-28T12:48:45.651099+0000","last_became_active":"2024-04-28T12:48:44.478121+0000","last_became_peered":"2024-04-28T12:48:44.478121+0000","last_unstale":"2024-04-28T12:48:45.651099+0000","last_undegraded":"2024-04-28T12:48:45.651099+0000","last_fullsized":"2024-04-28T12:48:45.651099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:07:43.195804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650390+0000","last_change":"2024-04-28T12:48:44.476475+0000","last_active":"2024-04-28T12:48:45.650390+0000","last_peered":"2024-04-28T12:48:45.650390+0000","last_clean":"2024-04-28T12:48:45.650390+0000","last_became_active":"2024-04-28T12:48:44.475839+0000","last_became_peered":"2024-04-28T12:48:44.475839+0000","last_unstale":"2024-04-28T12:48:45.650390+0000","last_undegraded":"2024-04-28T12:48:45.650390+0000","last_fullsized":"2024-04-28T12:48:45.650390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:34:51.513356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650117+0000","last_change":"2024-04-28T12:48:44.480569+0000","last_active":"2024-04-28T12:48:45.650117+0000","last_peered":"2024-04-28T12:48:45.650117+0000","last_clean":"2024-04-28T12:48:45.650117+0000","last_became_active":"2024-04-28T12:48:44.478085+0000","last_became_peered":"2024-04-28T12:48:44.478085+0000","last_unstale":"2024-04-28T12:48:45.650117+0000","last_undegraded":"2024-04-28T12:48:45.650117+0000","last_fullsized":"2024-04-28T12:48:45.650117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:23:13.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.649929+0000","last_change":"2024-04-28T12:48:44.477406+0000","last_active":"2024-04-28T12:48:45.649929+0000","last_peered":"2024-04-28T12:48:45.649929+0000","last_clean":"2024-04-28T12:48:45.649929+0000","last_became_active":"2024-04-28T12:48:44.477053+0000","last_became_peered":"2024-04-28T12:48:44.477053+0000","last_unstale":"2024-04-28T12:48:45.649929+0000","last_undegraded":"2024-04-28T12:48:45.649929+0000","last_fullsized":"2024-04-28T12:48:45.649929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:29.494688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650726+0000","last_change":"2024-04-28T12:48:44.485600+0000","last_active":"2024-04-28T12:48:45.650726+0000","last_peered":"2024-04-28T12:48:45.650726+0000","last_clean":"2024-04-28T12:48:45.650726+0000","last_became_active":"2024-04-28T12:48:44.485297+0000","last_became_peered":"2024-04-28T12:48:44.485297+0000","last_unstale":"2024-04-28T12:48:45.650726+0000","last_undegraded":"2024-04-28T12:48:45.650726+0000","last_fullsized":"2024-04-28T12:48:45.650726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:25:16.124798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.736089+0000","last_change":"2024-04-28T12:48:44.487376+0000","last_active":"2024-04-28T12:48:45.736089+0000","last_peered":"2024-04-28T12:48:45.736089+0000","last_clean":"2024-04-28T12:48:45.736089+0000","last_became_active":"2024-04-28T12:48:44.486561+0000","last_became_peered":"2024-04-28T12:48:44.486561+0000","last_unstale":"2024-04-28T12:48:45.736089+0000","last_undegraded":"2024-04-28T12:48:45.736089+0000","last_fullsized":"2024-04-28T12:48:45.736089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:43.398130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.519631+0000","last_change":"2024-04-28T12:48:44.475470+0000","last_active":"2024-04-28T12:48:45.519631+0000","last_peered":"2024-04-28T12:48:45.519631+0000","last_clean":"2024-04-28T12:48:45.519631+0000","last_became_active":"2024-04-28T12:48:44.475113+0000","last_became_peered":"2024-04-28T12:48:44.475113+0000","last_unstale":"2024-04-28T12:48:45.519631+0000","last_undegraded":"2024-04-28T12:48:45.519631+0000","last_fullsized":"2024-04-28T12:48:45.519631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:55:28.688458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650412+0000","last_change":"2024-04-28T12:48:44.482195+0000","last_active":"2024-04-28T12:48:45.650412+0000","last_peered":"2024-04-28T12:48:45.650412+0000","last_clean":"2024-04-28T12:48:45.650412+0000","last_became_active":"2024-04-28T12:48:44.481847+0000","last_became_peered":"2024-04-28T12:48:44.481847+0000","last_unstale":"2024-04-28T12:48:45.650412+0000","last_undegraded":"2024-04-28T12:48:45.650412+0000","last_fullsized":"2024-04-28T12:48:45.650412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:18:03.507012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.553256+0000","last_change":"2024-04-28T12:48:43.517808+0000","last_active":"2024-04-28T12:48:48.553256+0000","last_peered":"2024-04-28T12:48:48.553256+0000","last_clean":"2024-04-28T12:48:48.553256+0000","last_became_active":"2024-04-28T12:48:43.517443+0000","last_became_peered":"2024-04-28T12:48:43.517443+0000","last_unstale":"2024-04-28T12:48:48.553256+0000","last_undegraded":"2024-04-28T12:48:48.553256+0000","last_fullsized":"2024-04-28T12:48:48.553256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:57:11.972794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459687+0000","last_change":"2024-04-28T12:48:44.486296+0000","last_active":"2024-04-28T12:48:45.459687+0000","last_peered":"2024-04-28T12:48:45.459687+0000","last_clean":"2024-04-28T12:48:45.459687+0000","last_became_active":"2024-04-28T12:48:44.485289+0000","last_became_peered":"2024-04-28T12:48:44.485289+0000","last_unstale":"2024-04-28T12:48:45.459687+0000","last_undegraded":"2024-04-28T12:48:45.459687+0000","last_fullsized":"2024-04-28T12:48:45.459687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:07.558478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460213+0000","last_change":"2024-04-28T12:48:43.523725+0000","last_active":"2024-04-28T12:48:45.460213+0000","last_peered":"2024-04-28T12:48:45.460213+0000","last_clean":"2024-04-28T12:48:45.460213+0000","last_became_active":"2024-04-28T12:48:43.523367+0000","last_became_peered":"2024-04-28T12:48:43.523367+0000","last_unstale":"2024-04-28T12:48:45.460213+0000","last_undegraded":"2024-04-28T12:48:45.460213+0000","last_fullsized":"2024-04-28T12:48:45.460213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:15:15.674552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650922+0000","last_change":"2024-04-28T12:48:44.473393+0000","last_active":"2024-04-28T12:48:45.650922+0000","last_peered":"2024-04-28T12:48:45.650922+0000","last_clean":"2024-04-28T12:48:45.650922+0000","last_became_active":"2024-04-28T12:48:44.473035+0000","last_became_peered":"2024-04-28T12:48:44.473035+0000","last_unstale":"2024-04-28T12:48:45.650922+0000","last_undegraded":"2024-04-28T12:48:45.650922+0000","last_fullsized":"2024-04-28T12:48:45.650922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:06.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.550608+0000","last_change":"2024-04-28T12:48:43.510619+0000","last_active":"2024-04-28T12:48:48.550608+0000","last_peered":"2024-04-28T12:48:48.550608+0000","last_clean":"2024-04-28T12:48:48.550608+0000","last_became_active":"2024-04-28T12:48:43.510265+0000","last_became_peered":"2024-04-28T12:48:43.510265+0000","last_unstale":"2024-04-28T12:48:48.550608+0000","last_undegraded":"2024-04-28T12:48:48.550608+0000","last_fullsized":"2024-04-28T12:48:48.550608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:39:41.833527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651273+0000","last_change":"2024-04-28T12:48:44.475616+0000","last_active":"2024-04-28T12:48:45.651273+0000","last_peered":"2024-04-28T12:48:45.651273+0000","last_clean":"2024-04-28T12:48:45.651273+0000","last_became_active":"2024-04-28T12:48:44.475268+0000","last_became_peered":"2024-04-28T12:48:44.475268+0000","last_unstale":"2024-04-28T12:48:45.651273+0000","last_undegraded":"2024-04-28T12:48:45.651273+0000","last_fullsized":"2024-04-28T12:48:45.651273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:15:25.896749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.522012+0000","last_change":"2024-04-28T12:48:43.517652+0000","last_active":"2024-04-28T12:48:45.522012+0000","last_peered":"2024-04-28T12:48:45.522012+0000","last_clean":"2024-04-28T12:48:45.522012+0000","last_became_active":"2024-04-28T12:48:43.517220+0000","last_became_peered":"2024-04-28T12:48:43.517220+0000","last_unstale":"2024-04-28T12:48:45.522012+0000","last_undegraded":"2024-04-28T12:48:45.522012+0000","last_fullsized":"2024-04-28T12:48:45.522012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:25:52.793542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.461224+0000","last_change":"2024-04-28T12:48:44.488860+0000","last_active":"2024-04-28T12:48:45.461224+0000","last_peered":"2024-04-28T12:48:45.461224+0000","last_clean":"2024-04-28T12:48:45.461224+0000","last_became_active":"2024-04-28T12:48:44.488487+0000","last_became_peered":"2024-04-28T12:48:44.488487+0000","last_unstale":"2024-04-28T12:48:45.461224+0000","last_undegraded":"2024-04-28T12:48:45.461224+0000","last_fullsized":"2024-04-28T12:48:45.461224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:33:28.357280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543441+0000","last_change":"2024-04-28T12:48:43.507961+0000","last_active":"2024-04-28T12:48:47.543441+0000","last_peered":"2024-04-28T12:48:47.543441+0000","last_clean":"2024-04-28T12:48:47.543441+0000","last_became_active":"2024-04-28T12:48:43.507600+0000","last_became_peered":"2024-04-28T12:48:43.507600+0000","last_unstale":"2024-04-28T12:48:47.543441+0000","last_undegraded":"2024-04-28T12:48:47.543441+0000","last_fullsized":"2024-04-28T12:48:47.543441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:16.188154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521639+0000","last_change":"2024-04-28T12:48:44.474027+0000","last_active":"2024-04-28T12:48:45.521639+0000","last_peered":"2024-04-28T12:48:45.521639+0000","last_clean":"2024-04-28T12:48:45.521639+0000","last_became_active":"2024-04-28T12:48:44.473372+0000","last_became_peered":"2024-04-28T12:48:44.473372+0000","last_unstale":"2024-04-28T12:48:45.521639+0000","last_undegraded":"2024-04-28T12:48:45.521639+0000","last_fullsized":"2024-04-28T12:48:45.521639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:18:43.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.550171+0000","last_change":"2024-04-28T12:48:43.521135+0000","last_active":"2024-04-28T12:48:48.550171+0000","last_peered":"2024-04-28T12:48:48.550171+0000","last_clean":"2024-04-28T12:48:48.550171+0000","last_became_active":"2024-04-28T12:48:43.520749+0000","last_became_peered":"2024-04-28T12:48:43.520749+0000","last_unstale":"2024-04-28T12:48:48.550171+0000","last_undegraded":"2024-04-28T12:48:48.550171+0000","last_fullsized":"2024-04-28T12:48:48.550171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:35:33.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460966+0000","last_change":"2024-04-28T12:48:44.479090+0000","last_active":"2024-04-28T12:48:45.460966+0000","last_peered":"2024-04-28T12:48:45.460966+0000","last_clean":"2024-04-28T12:48:45.460966+0000","last_became_active":"2024-04-28T12:48:44.478725+0000","last_became_peered":"2024-04-28T12:48:44.478725+0000","last_unstale":"2024-04-28T12:48:45.460966+0000","last_undegraded":"2024-04-28T12:48:45.460966+0000","last_fullsized":"2024-04-28T12:48:45.460966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:40:21.710990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.550173+0000","last_change":"2024-04-28T12:48:43.505309+0000","last_active":"2024-04-28T12:48:48.550173+0000","last_peered":"2024-04-28T12:48:48.550173+0000","last_clean":"2024-04-28T12:48:48.550173+0000","last_became_active":"2024-04-28T12:48:43.504934+0000","last_became_peered":"2024-04-28T12:48:43.504934+0000","last_unstale":"2024-04-28T12:48:48.550173+0000","last_undegraded":"2024-04-28T12:48:48.550173+0000","last_fullsized":"2024-04-28T12:48:48.550173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:48:39.652606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459557+0000","last_change":"2024-04-28T12:48:44.479930+0000","last_active":"2024-04-28T12:48:45.459557+0000","last_peered":"2024-04-28T12:48:45.459557+0000","last_clean":"2024-04-28T12:48:45.459557+0000","last_became_active":"2024-04-28T12:48:44.479571+0000","last_became_peered":"2024-04-28T12:48:44.479571+0000","last_unstale":"2024-04-28T12:48:45.459557+0000","last_undegraded":"2024-04-28T12:48:45.459557+0000","last_fullsized":"2024-04-28T12:48:45.459557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:00.130664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460641+0000","last_change":"2024-04-28T12:48:43.508935+0000","last_active":"2024-04-28T12:48:45.460641+0000","last_peered":"2024-04-28T12:48:45.460641+0000","last_clean":"2024-04-28T12:48:45.460641+0000","last_became_active":"2024-04-28T12:48:43.508596+0000","last_became_peered":"2024-04-28T12:48:43.508596+0000","last_unstale":"2024-04-28T12:48:45.460641+0000","last_undegraded":"2024-04-28T12:48:45.460641+0000","last_fullsized":"2024-04-28T12:48:45.460641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:41.905528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.519419+0000","last_change":"2024-04-28T12:48:44.489695+0000","last_active":"2024-04-28T12:48:45.519419+0000","last_peered":"2024-04-28T12:48:45.519419+0000","last_clean":"2024-04-28T12:48:45.519419+0000","last_became_active":"2024-04-28T12:48:44.489290+0000","last_became_peered":"2024-04-28T12:48:44.489290+0000","last_unstale":"2024-04-28T12:48:45.519419+0000","last_undegraded":"2024-04-28T12:48:45.519419+0000","last_fullsized":"2024-04-28T12:48:45.519419+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:34:01.873775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543827+0000","last_change":"2024-04-28T12:48:43.511751+0000","last_active":"2024-04-28T12:48:47.543827+0000","last_peered":"2024-04-28T12:48:47.543827+0000","last_clean":"2024-04-28T12:48:47.543827+0000","last_became_active":"2024-04-28T12:48:43.511428+0000","last_became_peered":"2024-04-28T12:48:43.511428+0000","last_unstale":"2024-04-28T12:48:47.543827+0000","last_undegraded":"2024-04-28T12:48:47.543827+0000","last_fullsized":"2024-04-28T12:48:47.543827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:12.987697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651765+0000","last_change":"2024-04-28T12:48:44.472310+0000","last_active":"2024-04-28T12:48:45.651765+0000","last_peered":"2024-04-28T12:48:45.651765+0000","last_clean":"2024-04-28T12:48:45.651765+0000","last_became_active":"2024-04-28T12:48:44.471978+0000","last_became_peered":"2024-04-28T12:48:44.471978+0000","last_unstale":"2024-04-28T12:48:45.651765+0000","last_undegraded":"2024-04-28T12:48:45.651765+0000","last_fullsized":"2024-04-28T12:48:45.651765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.068156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.542891+0000","last_change":"2024-04-28T12:48:43.514475+0000","last_active":"2024-04-28T12:48:47.542891+0000","last_peered":"2024-04-28T12:48:47.542891+0000","last_clean":"2024-04-28T12:48:47.542891+0000","last_became_active":"2024-04-28T12:48:43.514067+0000","last_became_peered":"2024-04-28T12:48:43.514067+0000","last_unstale":"2024-04-28T12:48:47.542891+0000","last_undegraded":"2024-04-28T12:48:47.542891+0000","last_fullsized":"2024-04-28T12:48:47.542891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:52.125456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650790+0000","last_change":"2024-04-28T12:48:44.479210+0000","last_active":"2024-04-28T12:48:45.650790+0000","last_peered":"2024-04-28T12:48:45.650790+0000","last_clean":"2024-04-28T12:48:45.650790+0000","last_became_active":"2024-04-28T12:48:44.478816+0000","last_became_peered":"2024-04-28T12:48:44.478816+0000","last_unstale":"2024-04-28T12:48:45.650790+0000","last_undegraded":"2024-04-28T12:48:45.650790+0000","last_fullsized":"2024-04-28T12:48:45.650790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:03:06.994063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.461165+0000","last_change":"2024-04-28T12:48:43.520207+0000","last_active":"2024-04-28T12:48:45.461165+0000","last_peered":"2024-04-28T12:48:45.461165+0000","last_clean":"2024-04-28T12:48:45.461165+0000","last_became_active":"2024-04-28T12:48:43.519823+0000","last_became_peered":"2024-04-28T12:48:43.519823+0000","last_unstale":"2024-04-28T12:48:45.461165+0000","last_undegraded":"2024-04-28T12:48:45.461165+0000","last_fullsized":"2024-04-28T12:48:45.461165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:02.238166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651160+0000","last_change":"2024-04-28T12:48:44.479632+0000","last_active":"2024-04-28T12:48:45.651160+0000","last_peered":"2024-04-28T12:48:45.651160+0000","last_clean":"2024-04-28T12:48:45.651160+0000","last_became_active":"2024-04-28T12:48:44.479197+0000","last_became_peered":"2024-04-28T12:48:44.479197+0000","last_unstale":"2024-04-28T12:48:45.651160+0000","last_undegraded":"2024-04-28T12:48:45.651160+0000","last_fullsized":"2024-04-28T12:48:45.651160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:49:01.557312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543994+0000","last_change":"2024-04-28T12:48:43.514198+0000","last_active":"2024-04-28T12:48:47.543994+0000","last_peered":"2024-04-28T12:48:47.543994+0000","last_clean":"2024-04-28T12:48:47.543994+0000","last_became_active":"2024-04-28T12:48:43.513839+0000","last_became_peered":"2024-04-28T12:48:43.513839+0000","last_unstale":"2024-04-28T12:48:47.543994+0000","last_undegraded":"2024-04-28T12:48:47.543994+0000","last_fullsized":"2024-04-28T12:48:47.543994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:54:19.446610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521856+0000","last_change":"2024-04-28T12:48:44.484996+0000","last_active":"2024-04-28T12:48:45.521856+0000","last_peered":"2024-04-28T12:48:45.521856+0000","last_clean":"2024-04-28T12:48:45.521856+0000","last_became_active":"2024-04-28T12:48:44.484591+0000","last_became_peered":"2024-04-28T12:48:44.484591+0000","last_unstale":"2024-04-28T12:48:45.521856+0000","last_undegraded":"2024-04-28T12:48:45.521856+0000","last_fullsized":"2024-04-28T12:48:45.521856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:38:42.768757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.552644+0000","last_change":"2024-04-28T12:48:43.510365+0000","last_active":"2024-04-28T12:48:48.552644+0000","last_peered":"2024-04-28T12:48:48.552644+0000","last_clean":"2024-04-28T12:48:48.552644+0000","last_became_active":"2024-04-28T12:48:43.510028+0000","last_became_peered":"2024-04-28T12:48:43.510028+0000","last_unstale":"2024-04-28T12:48:48.552644+0000","last_undegraded":"2024-04-28T12:48:48.552644+0000","last_fullsized":"2024-04-28T12:48:48.552644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:09:20.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650200+0000","last_change":"2024-04-28T12:48:44.489704+0000","last_active":"2024-04-28T12:48:45.650200+0000","last_peered":"2024-04-28T12:48:45.650200+0000","last_clean":"2024-04-28T12:48:45.650200+0000","last_became_active":"2024-04-28T12:48:44.489331+0000","last_became_peered":"2024-04-28T12:48:44.489331+0000","last_unstale":"2024-04-28T12:48:45.650200+0000","last_undegraded":"2024-04-28T12:48:45.650200+0000","last_fullsized":"2024-04-28T12:48:45.650200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:33.755818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.542937+0000","last_change":"2024-04-28T12:48:43.519213+0000","last_active":"2024-04-28T12:48:47.542937+0000","last_peered":"2024-04-28T12:48:47.542937+0000","last_clean":"2024-04-28T12:48:47.542937+0000","last_became_active":"2024-04-28T12:48:43.518385+0000","last_became_peered":"2024-04-28T12:48:43.518385+0000","last_unstale":"2024-04-28T12:48:47.542937+0000","last_undegraded":"2024-04-28T12:48:47.542937+0000","last_fullsized":"2024-04-28T12:48:47.542937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:31:37.130504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463742+0000","last_change":"2024-04-28T12:48:44.480838+0000","last_active":"2024-04-28T12:48:45.463742+0000","last_peered":"2024-04-28T12:48:45.463742+0000","last_clean":"2024-04-28T12:48:45.463742+0000","last_became_active":"2024-04-28T12:48:44.480522+0000","last_became_peered":"2024-04-28T12:48:44.480522+0000","last_unstale":"2024-04-28T12:48:45.463742+0000","last_undegraded":"2024-04-28T12:48:45.463742+0000","last_fullsized":"2024-04-28T12:48:45.463742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:24.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:49.467014+0000","last_change":"2024-04-28T12:48:43.505498+0000","last_active":"2024-04-28T12:48:49.467014+0000","last_peered":"2024-04-28T12:48:49.467014+0000","last_clean":"2024-04-28T12:48:49.467014+0000","last_became_active":"2024-04-28T12:48:43.505041+0000","last_became_peered":"2024-04-28T12:48:43.505041+0000","last_unstale":"2024-04-28T12:48:49.467014+0000","last_undegraded":"2024-04-28T12:48:49.467014+0000","last_fullsized":"2024-04-28T12:48:49.467014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:36.624508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463192+0000","last_change":"2024-04-28T12:48:44.482698+0000","last_active":"2024-04-28T12:48:45.463192+0000","last_peered":"2024-04-28T12:48:45.463192+0000","last_clean":"2024-04-28T12:48:45.463192+0000","last_became_active":"2024-04-28T12:48:44.482365+0000","last_became_peered":"2024-04-28T12:48:44.482365+0000","last_unstale":"2024-04-28T12:48:45.463192+0000","last_undegraded":"2024-04-28T12:48:45.463192+0000","last_fullsized":"2024-04-28T12:48:45.463192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:45:33.648105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.552967+0000","last_change":"2024-04-28T12:48:43.515942+0000","last_active":"2024-04-28T12:48:48.552967+0000","last_peered":"2024-04-28T12:48:48.552967+0000","last_clean":"2024-04-28T12:48:48.552967+0000","last_became_active":"2024-04-28T12:48:43.515497+0000","last_became_peered":"2024-04-28T12:48:43.515497+0000","last_unstale":"2024-04-28T12:48:48.552967+0000","last_undegraded":"2024-04-28T12:48:48.552967+0000","last_fullsized":"2024-04-28T12:48:48.552967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:39.489486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460566+0000","last_change":"2024-04-28T12:48:44.487831+0000","last_active":"2024-04-28T12:48:45.460566+0000","last_peered":"2024-04-28T12:48:45.460566+0000","last_clean":"2024-04-28T12:48:45.460566+0000","last_became_active":"2024-04-28T12:48:44.487341+0000","last_became_peered":"2024-04-28T12:48:44.487341+0000","last_unstale":"2024-04-28T12:48:45.460566+0000","last_undegraded":"2024-04-28T12:48:45.460566+0000","last_fullsized":"2024-04-28T12:48:45.460566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:42.810514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:49.462638+0000","last_change":"2024-04-28T12:48:43.513667+0000","last_active":"2024-04-28T12:48:49.462638+0000","last_peered":"2024-04-28T12:48:49.462638+0000","last_clean":"2024-04-28T12:48:49.462638+0000","last_became_active":"2024-04-28T12:48:43.513192+0000","last_became_peered":"2024-04-28T12:48:43.513192+0000","last_unstale":"2024-04-28T12:48:49.462638+0000","last_undegraded":"2024-04-28T12:48:49.462638+0000","last_fullsized":"2024-04-28T12:48:49.462638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.470320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.735736+0000","last_change":"2024-04-28T12:48:44.478203+0000","last_active":"2024-04-28T12:48:45.735736+0000","last_peered":"2024-04-28T12:48:45.735736+0000","last_clean":"2024-04-28T12:48:45.735736+0000","last_became_active":"2024-04-28T12:48:44.476238+0000","last_became_peered":"2024-04-28T12:48:44.476238+0000","last_unstale":"2024-04-28T12:48:45.735736+0000","last_undegraded":"2024-04-28T12:48:45.735736+0000","last_fullsized":"2024-04-28T12:48:45.735736+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:46:49.810015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.519327+0000","last_change":"2024-04-28T12:48:43.516036+0000","last_active":"2024-04-28T12:48:45.519327+0000","last_peered":"2024-04-28T12:48:45.519327+0000","last_clean":"2024-04-28T12:48:45.519327+0000","last_became_active":"2024-04-28T12:48:43.515494+0000","last_became_peered":"2024-04-28T12:48:43.515494+0000","last_unstale":"2024-04-28T12:48:45.519327+0000","last_undegraded":"2024-04-28T12:48:45.519327+0000","last_fullsized":"2024-04-28T12:48:45.519327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:34:56.568179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651243+0000","last_change":"2024-04-28T12:48:44.476836+0000","last_active":"2024-04-28T12:48:45.651243+0000","last_peered":"2024-04-28T12:48:45.651243+0000","last_clean":"2024-04-28T12:48:45.651243+0000","last_became_active":"2024-04-28T12:48:44.476214+0000","last_became_peered":"2024-04-28T12:48:44.476214+0000","last_unstale":"2024-04-28T12:48:45.651243+0000","last_undegraded":"2024-04-28T12:48:45.651243+0000","last_fullsized":"2024-04-28T12:48:45.651243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:24:26.707985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:46.870585+0000","last_change":"2024-04-28T12:48:43.520918+0000","last_active":"2024-04-28T12:48:46.870585+0000","last_peered":"2024-04-28T12:48:46.870585+0000","last_clean":"2024-04-28T12:48:46.870585+0000","last_became_active":"2024-04-28T12:48:43.520564+0000","last_became_peered":"2024-04-28T12:48:43.520564+0000","last_unstale":"2024-04-28T12:48:46.870585+0000","last_undegraded":"2024-04-28T12:48:46.870585+0000","last_fullsized":"2024-04-28T12:48:46.870585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:32:51.206102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650906+0000","last_change":"2024-04-28T12:48:44.484570+0000","last_active":"2024-04-28T12:48:45.650906+0000","last_peered":"2024-04-28T12:48:45.650906+0000","last_clean":"2024-04-28T12:48:45.650906+0000","last_became_active":"2024-04-28T12:48:44.484133+0000","last_became_peered":"2024-04-28T12:48:44.484133+0000","last_unstale":"2024-04-28T12:48:45.650906+0000","last_undegraded":"2024-04-28T12:48:45.650906+0000","last_fullsized":"2024-04-28T12:48:45.650906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:59:14.274873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460829+0000","last_change":"2024-04-28T12:48:44.478013+0000","last_active":"2024-04-28T12:48:45.460829+0000","last_peered":"2024-04-28T12:48:45.460829+0000","last_clean":"2024-04-28T12:48:45.460829+0000","last_became_active":"2024-04-28T12:48:44.477559+0000","last_became_peered":"2024-04-28T12:48:44.477559+0000","last_unstale":"2024-04-28T12:48:45.460829+0000","last_undegraded":"2024-04-28T12:48:45.460829+0000","last_fullsized":"2024-04-28T12:48:45.460829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:57:49.931865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:49.464459+0000","last_change":"2024-04-28T12:48:43.515873+0000","last_active":"2024-04-28T12:48:49.464459+0000","last_peered":"2024-04-28T12:48:49.464459+0000","last_clean":"2024-04-28T12:48:49.464459+0000","last_became_active":"2024-04-28T12:48:43.515442+0000","last_became_peered":"2024-04-28T12:48:43.515442+0000","last_unstale":"2024-04-28T12:48:49.464459+0000","last_undegraded":"2024-04-28T12:48:49.464459+0000","last_fullsized":"2024-04-28T12:48:49.464459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:29:57.973433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459795+0000","last_change":"2024-04-28T12:48:44.479868+0000","last_active":"2024-04-28T12:48:45.459795+0000","last_peered":"2024-04-28T12:48:45.459795+0000","last_clean":"2024-04-28T12:48:45.459795+0000","last_became_active":"2024-04-28T12:48:44.479519+0000","last_became_peered":"2024-04-28T12:48:44.479519+0000","last_unstale":"2024-04-28T12:48:45.459795+0000","last_undegraded":"2024-04-28T12:48:45.459795+0000","last_fullsized":"2024-04-28T12:48:45.459795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:09:20.946380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:49.463136+0000","last_change":"2024-04-28T12:48:43.507662+0000","last_active":"2024-04-28T12:48:49.463136+0000","last_peered":"2024-04-28T12:48:49.463136+0000","last_clean":"2024-04-28T12:48:49.463136+0000","last_became_active":"2024-04-28T12:48:43.507331+0000","last_became_peered":"2024-04-28T12:48:43.507331+0000","last_unstale":"2024-04-28T12:48:49.463136+0000","last_undegraded":"2024-04-28T12:48:49.463136+0000","last_fullsized":"2024-04-28T12:48:49.463136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:10.073569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":338,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463187+0000","last_change":"2024-04-28T12:47:27.444407+0000","last_active":"2024-04-28T12:48:45.463187+0000","last_peered":"2024-04-28T12:48:45.463187+0000","last_clean":"2024-04-28T12:48:45.463187+0000","last_became_active":"2024-04-28T12:47:27.443992+0000","last_became_peered":"2024-04-28T12:47:27.443992+0000","last_unstale":"2024-04-28T12:48:45.463187+0000","last_undegraded":"2024-04-28T12:48:45.463187+0000","last_fullsized":"2024-04-28T12:48:45.463187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_clean_scrub_stamp":"2024-04-28T12:44:00.636011+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:17:23.614524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.649824+0000","last_change":"2024-04-28T12:48:44.479395+0000","last_active":"2024-04-28T12:48:45.649824+0000","last_peered":"2024-04-28T12:48:45.649824+0000","last_clean":"2024-04-28T12:48:45.649824+0000","last_became_active":"2024-04-28T12:48:44.478935+0000","last_became_peered":"2024-04-28T12:48:44.478935+0000","last_unstale":"2024-04-28T12:48:45.649824+0000","last_undegraded":"2024-04-28T12:48:45.649824+0000","last_fullsized":"2024-04-28T12:48:45.649824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:19:28.212744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.549972+0000","last_change":"2024-04-28T12:48:43.512026+0000","last_active":"2024-04-28T12:48:48.549972+0000","last_peered":"2024-04-28T12:48:48.549972+0000","last_clean":"2024-04-28T12:48:48.549972+0000","last_became_active":"2024-04-28T12:48:43.511650+0000","last_became_peered":"2024-04-28T12:48:43.511650+0000","last_unstale":"2024-04-28T12:48:48.549972+0000","last_undegraded":"2024-04-28T12:48:48.549972+0000","last_fullsized":"2024-04-28T12:48:48.549972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:38:04.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.542735+0000","last_change":"2024-04-28T12:48:43.520068+0000","last_active":"2024-04-28T12:48:47.542735+0000","last_peered":"2024-04-28T12:48:47.542735+0000","last_clean":"2024-04-28T12:48:47.542735+0000","last_became_active":"2024-04-28T12:48:43.519624+0000","last_became_peered":"2024-04-28T12:48:43.519624+0000","last_unstale":"2024-04-28T12:48:47.542735+0000","last_undegraded":"2024-04-28T12:48:47.542735+0000","last_fullsized":"2024-04-28T12:48:47.542735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:35:45.942437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650390+0000","last_change":"2024-04-28T12:48:44.470458+0000","last_active":"2024-04-28T12:48:45.650390+0000","last_peered":"2024-04-28T12:48:45.650390+0000","last_clean":"2024-04-28T12:48:45.650390+0000","last_became_active":"2024-04-28T12:48:44.470105+0000","last_became_peered":"2024-04-28T12:48:44.470105+0000","last_unstale":"2024-04-28T12:48:45.650390+0000","last_undegraded":"2024-04-28T12:48:45.650390+0000","last_fullsized":"2024-04-28T12:48:45.650390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:49:25.970990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.551971+0000","last_change":"2024-04-28T12:48:43.510065+0000","last_active":"2024-04-28T12:48:48.551971+0000","last_peered":"2024-04-28T12:48:48.551971+0000","last_clean":"2024-04-28T12:48:48.551971+0000","last_became_active":"2024-04-28T12:48:43.509718+0000","last_became_peered":"2024-04-28T12:48:43.509718+0000","last_unstale":"2024-04-28T12:48:48.551971+0000","last_undegraded":"2024-04-28T12:48:48.551971+0000","last_fullsized":"2024-04-28T12:48:48.551971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:50:46.776100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459879+0000","last_change":"2024-04-28T12:48:44.487022+0000","last_active":"2024-04-28T12:48:45.459879+0000","last_peered":"2024-04-28T12:48:45.459879+0000","last_clean":"2024-04-28T12:48:45.459879+0000","last_became_active":"2024-04-28T12:48:44.486675+0000","last_became_peered":"2024-04-28T12:48:44.486675+0000","last_unstale":"2024-04-28T12:48:45.459879+0000","last_undegraded":"2024-04-28T12:48:45.459879+0000","last_fullsized":"2024-04-28T12:48:45.459879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:36.185845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543383+0000","last_change":"2024-04-28T12:48:43.505464+0000","last_active":"2024-04-28T12:48:47.543383+0000","last_peered":"2024-04-28T12:48:47.543383+0000","last_clean":"2024-04-28T12:48:47.543383+0000","last_became_active":"2024-04-28T12:48:43.505085+0000","last_became_peered":"2024-04-28T12:48:43.505085+0000","last_unstale":"2024-04-28T12:48:47.543383+0000","last_undegraded":"2024-04-28T12:48:47.543383+0000","last_fullsized":"2024-04-28T12:48:47.543383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:37:47.875243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650740+0000","last_change":"2024-04-28T12:48:44.486517+0000","last_active":"2024-04-28T12:48:45.650740+0000","last_peered":"2024-04-28T12:48:45.650740+0000","last_clean":"2024-04-28T12:48:45.650740+0000","last_became_active":"2024-04-28T12:48:44.486139+0000","last_became_peered":"2024-04-28T12:48:44.486139+0000","last_unstale":"2024-04-28T12:48:45.650740+0000","last_undegraded":"2024-04-28T12:48:45.650740+0000","last_fullsized":"2024-04-28T12:48:45.650740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:32:23.762496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.542615+0000","last_change":"2024-04-28T12:48:43.508992+0000","last_active":"2024-04-28T12:48:47.542615+0000","last_peered":"2024-04-28T12:48:47.542615+0000","last_clean":"2024-04-28T12:48:47.542615+0000","last_became_active":"2024-04-28T12:48:43.508492+0000","last_became_peered":"2024-04-28T12:48:43.508492+0000","last_unstale":"2024-04-28T12:48:47.542615+0000","last_undegraded":"2024-04-28T12:48:47.542615+0000","last_fullsized":"2024-04-28T12:48:47.542615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:56:29.138232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.521559+0000","last_change":"2024-04-28T12:48:44.476462+0000","last_active":"2024-04-28T12:48:45.521559+0000","last_peered":"2024-04-28T12:48:45.521559+0000","last_clean":"2024-04-28T12:48:45.521559+0000","last_became_active":"2024-04-28T12:48:44.475660+0000","last_became_peered":"2024-04-28T12:48:44.475660+0000","last_unstale":"2024-04-28T12:48:45.521559+0000","last_undegraded":"2024-04-28T12:48:45.521559+0000","last_fullsized":"2024-04-28T12:48:45.521559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:49:48.656837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.542424+0000","last_change":"2024-04-28T12:48:43.508946+0000","last_active":"2024-04-28T12:48:47.542424+0000","last_peered":"2024-04-28T12:48:47.542424+0000","last_clean":"2024-04-28T12:48:47.542424+0000","last_became_active":"2024-04-28T12:48:43.508553+0000","last_became_peered":"2024-04-28T12:48:43.508553+0000","last_unstale":"2024-04-28T12:48:47.542424+0000","last_undegraded":"2024-04-28T12:48:47.542424+0000","last_fullsized":"2024-04-28T12:48:47.542424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:52:56.461509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.735662+0000","last_change":"2024-04-28T12:48:44.473967+0000","last_active":"2024-04-28T12:48:45.735662+0000","last_peered":"2024-04-28T12:48:45.735662+0000","last_clean":"2024-04-28T12:48:45.735662+0000","last_became_active":"2024-04-28T12:48:44.473614+0000","last_became_peered":"2024-04-28T12:48:44.473614+0000","last_unstale":"2024-04-28T12:48:45.735662+0000","last_undegraded":"2024-04-28T12:48:45.735662+0000","last_fullsized":"2024-04-28T12:48:45.735662+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:58:20.994924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650347+0000","last_change":"2024-04-28T12:48:43.511287+0000","last_active":"2024-04-28T12:48:45.650347+0000","last_peered":"2024-04-28T12:48:45.650347+0000","last_clean":"2024-04-28T12:48:45.650347+0000","last_became_active":"2024-04-28T12:48:43.510955+0000","last_became_peered":"2024-04-28T12:48:43.510955+0000","last_unstale":"2024-04-28T12:48:45.650347+0000","last_undegraded":"2024-04-28T12:48:45.650347+0000","last_fullsized":"2024-04-28T12:48:45.650347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:40.952405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459880+0000","last_change":"2024-04-28T12:48:44.480514+0000","last_active":"2024-04-28T12:48:45.459880+0000","last_peered":"2024-04-28T12:48:45.459880+0000","last_clean":"2024-04-28T12:48:45.459880+0000","last_became_active":"2024-04-28T12:48:44.480160+0000","last_became_peered":"2024-04-28T12:48:44.480160+0000","last_unstale":"2024-04-28T12:48:45.459880+0000","last_undegraded":"2024-04-28T12:48:45.459880+0000","last_fullsized":"2024-04-28T12:48:45.459880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:07:02.221492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:47.543651+0000","last_change":"2024-04-28T12:48:43.511823+0000","last_active":"2024-04-28T12:48:47.543651+0000","last_peered":"2024-04-28T12:48:47.543651+0000","last_clean":"2024-04-28T12:48:47.543651+0000","last_became_active":"2024-04-28T12:48:43.511420+0000","last_became_peered":"2024-04-28T12:48:43.511420+0000","last_unstale":"2024-04-28T12:48:47.543651+0000","last_undegraded":"2024-04-28T12:48:47.543651+0000","last_fullsized":"2024-04-28T12:48:47.543651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:59:10.170003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463551+0000","last_change":"2024-04-28T12:48:44.474095+0000","last_active":"2024-04-28T12:48:45.463551+0000","last_peered":"2024-04-28T12:48:45.463551+0000","last_clean":"2024-04-28T12:48:45.463551+0000","last_became_active":"2024-04-28T12:48:44.473377+0000","last_became_peered":"2024-04-28T12:48:44.473377+0000","last_unstale":"2024-04-28T12:48:45.463551+0000","last_undegraded":"2024-04-28T12:48:45.463551+0000","last_fullsized":"2024-04-28T12:48:45.463551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.355527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:48.552770+0000","last_change":"2024-04-28T12:48:43.515654+0000","last_active":"2024-04-28T12:48:48.552770+0000","last_peered":"2024-04-28T12:48:48.552770+0000","last_clean":"2024-04-28T12:48:48.552770+0000","last_became_active":"2024-04-28T12:48:43.515056+0000","last_became_peered":"2024-04-28T12:48:43.515056+0000","last_unstale":"2024-04-28T12:48:48.552770+0000","last_undegraded":"2024-04-28T12:48:48.552770+0000","last_fullsized":"2024-04-28T12:48:48.552770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:47:58.297248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463982+0000","last_change":"2024-04-28T12:48:44.474106+0000","last_active":"2024-04-28T12:48:45.463982+0000","last_peered":"2024-04-28T12:48:45.463982+0000","last_clean":"2024-04-28T12:48:45.463982+0000","last_became_active":"2024-04-28T12:48:44.473625+0000","last_became_peered":"2024-04-28T12:48:44.473625+0000","last_unstale":"2024-04-28T12:48:45.463982+0000","last_undegraded":"2024-04-28T12:48:45.463982+0000","last_fullsized":"2024-04-28T12:48:45.463982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:28:44.434155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460973+0000","last_change":"2024-04-28T12:48:43.519658+0000","last_active":"2024-04-28T12:48:45.460973+0000","last_peered":"2024-04-28T12:48:45.460973+0000","last_clean":"2024-04-28T12:48:45.460973+0000","last_became_active":"2024-04-28T12:48:43.519291+0000","last_became_peered":"2024-04-28T12:48:43.519291+0000","last_unstale":"2024-04-28T12:48:45.460973+0000","last_undegraded":"2024-04-28T12:48:45.460973+0000","last_fullsized":"2024-04-28T12:48:45.460973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:41.356834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:46.870615+0000","last_change":"2024-04-28T12:48:44.483790+0000","last_active":"2024-04-28T12:48:46.870615+0000","last_peered":"2024-04-28T12:48:46.870615+0000","last_clean":"2024-04-28T12:48:46.870615+0000","last_became_active":"2024-04-28T12:48:44.483442+0000","last_became_peered":"2024-04-28T12:48:44.483442+0000","last_unstale":"2024-04-28T12:48:46.870615+0000","last_undegraded":"2024-04-28T12:48:46.870615+0000","last_fullsized":"2024-04-28T12:48:46.870615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:38:59.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650474+0000","last_change":"2024-04-28T12:48:43.505650+0000","last_active":"2024-04-28T12:48:45.650474+0000","last_peered":"2024-04-28T12:48:45.650474+0000","last_clean":"2024-04-28T12:48:45.650474+0000","last_became_active":"2024-04-28T12:48:43.505305+0000","last_became_peered":"2024-04-28T12:48:43.505305+0000","last_unstale":"2024-04-28T12:48:45.650474+0000","last_undegraded":"2024-04-28T12:48:45.650474+0000","last_fullsized":"2024-04-28T12:48:45.650474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:56:34.654105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.463834+0000","last_change":"2024-04-28T12:48:44.478335+0000","last_active":"2024-04-28T12:48:45.463834+0000","last_peered":"2024-04-28T12:48:45.463834+0000","last_clean":"2024-04-28T12:48:45.463834+0000","last_became_active":"2024-04-28T12:48:44.477958+0000","last_became_peered":"2024-04-28T12:48:44.477958+0000","last_unstale":"2024-04-28T12:48:45.463834+0000","last_undegraded":"2024-04-28T12:48:45.463834+0000","last_fullsized":"2024-04-28T12:48:45.463834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:27.168011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.735829+0000","last_change":"2024-04-28T12:48:44.479601+0000","last_active":"2024-04-28T12:48:45.735829+0000","last_peered":"2024-04-28T12:48:45.735829+0000","last_clean":"2024-04-28T12:48:45.735829+0000","last_became_active":"2024-04-28T12:48:44.479262+0000","last_became_peered":"2024-04-28T12:48:44.479262+0000","last_unstale":"2024-04-28T12:48:45.735829+0000","last_undegraded":"2024-04-28T12:48:45.735829+0000","last_fullsized":"2024-04-28T12:48:45.735829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:14:56.808866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650168+0000","last_change":"2024-04-28T12:48:44.478553+0000","last_active":"2024-04-28T12:48:45.650168+0000","last_peered":"2024-04-28T12:48:45.650168+0000","last_clean":"2024-04-28T12:48:45.650168+0000","last_became_active":"2024-04-28T12:48:44.477539+0000","last_became_peered":"2024-04-28T12:48:44.477539+0000","last_unstale":"2024-04-28T12:48:45.650168+0000","last_undegraded":"2024-04-28T12:48:45.650168+0000","last_fullsized":"2024-04-28T12:48:45.650168+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:03:44.149337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650143+0000","last_change":"2024-04-28T12:48:44.483534+0000","last_active":"2024-04-28T12:48:45.650143+0000","last_peered":"2024-04-28T12:48:45.650143+0000","last_clean":"2024-04-28T12:48:45.650143+0000","last_became_active":"2024-04-28T12:48:44.483155+0000","last_became_peered":"2024-04-28T12:48:44.483155+0000","last_unstale":"2024-04-28T12:48:45.650143+0000","last_undegraded":"2024-04-28T12:48:45.650143+0000","last_fullsized":"2024-04-28T12:48:45.650143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:28:39.903443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650285+0000","last_change":"2024-04-28T12:48:44.479729+0000","last_active":"2024-04-28T12:48:45.650285+0000","last_peered":"2024-04-28T12:48:45.650285+0000","last_clean":"2024-04-28T12:48:45.650285+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:48:45.650285+0000","last_undegraded":"2024-04-28T12:48:45.650285+0000","last_fullsized":"2024-04-28T12:48:45.650285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:33:32.456549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650595+0000","last_change":"2024-04-28T12:48:44.486785+0000","last_active":"2024-04-28T12:48:45.650595+0000","last_peered":"2024-04-28T12:48:45.650595+0000","last_clean":"2024-04-28T12:48:45.650595+0000","last_became_active":"2024-04-28T12:48:44.485634+0000","last_became_peered":"2024-04-28T12:48:44.485634+0000","last_unstale":"2024-04-28T12:48:45.650595+0000","last_undegraded":"2024-04-28T12:48:45.650595+0000","last_fullsized":"2024-04-28T12:48:45.650595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:58.148545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.735667+0000","last_change":"2024-04-28T12:48:44.479690+0000","last_active":"2024-04-28T12:48:45.735667+0000","last_peered":"2024-04-28T12:48:45.735667+0000","last_clean":"2024-04-28T12:48:45.735667+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:48:45.735667+0000","last_undegraded":"2024-04-28T12:48:45.735667+0000","last_fullsized":"2024-04-28T12:48:45.735667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:40:47.274081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650565+0000","last_change":"2024-04-28T12:48:44.472305+0000","last_active":"2024-04-28T12:48:45.650565+0000","last_peered":"2024-04-28T12:48:45.650565+0000","last_clean":"2024-04-28T12:48:45.650565+0000","last_became_active":"2024-04-28T12:48:44.471227+0000","last_became_peered":"2024-04-28T12:48:44.471227+0000","last_unstale":"2024-04-28T12:48:45.650565+0000","last_undegraded":"2024-04-28T12:48:45.650565+0000","last_fullsized":"2024-04-28T12:48:45.650565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:27:14.602075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459670+0000","last_change":"2024-04-28T12:48:44.490866+0000","last_active":"2024-04-28T12:48:45.459670+0000","last_peered":"2024-04-28T12:48:45.459670+0000","last_clean":"2024-04-28T12:48:45.459670+0000","last_became_active":"2024-04-28T12:48:44.490515+0000","last_became_peered":"2024-04-28T12:48:44.490515+0000","last_unstale":"2024-04-28T12:48:45.459670+0000","last_undegraded":"2024-04-28T12:48:45.459670+0000","last_fullsized":"2024-04-28T12:48:45.459670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:19:22.404853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.460659+0000","last_change":"2024-04-28T12:48:44.480611+0000","last_active":"2024-04-28T12:48:45.460659+0000","last_peered":"2024-04-28T12:48:45.460659+0000","last_clean":"2024-04-28T12:48:45.460659+0000","last_became_active":"2024-04-28T12:48:44.480294+0000","last_became_peered":"2024-04-28T12:48:44.480294+0000","last_unstale":"2024-04-28T12:48:45.460659+0000","last_undegraded":"2024-04-28T12:48:45.460659+0000","last_fullsized":"2024-04-28T12:48:45.460659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:55.450825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.459582+0000","last_change":"2024-04-28T12:48:44.483003+0000","last_active":"2024-04-28T12:48:45.459582+0000","last_peered":"2024-04-28T12:48:45.459582+0000","last_clean":"2024-04-28T12:48:45.459582+0000","last_became_active":"2024-04-28T12:48:44.482630+0000","last_became_peered":"2024-04-28T12:48:44.482630+0000","last_unstale":"2024-04-28T12:48:45.459582+0000","last_undegraded":"2024-04-28T12:48:45.459582+0000","last_fullsized":"2024-04-28T12:48:45.459582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:05:58.358655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.651078+0000","last_change":"2024-04-28T12:48:44.489780+0000","last_active":"2024-04-28T12:48:45.651078+0000","last_peered":"2024-04-28T12:48:45.651078+0000","last_clean":"2024-04-28T12:48:45.651078+0000","last_became_active":"2024-04-28T12:48:44.489276+0000","last_became_peered":"2024-04-28T12:48:44.489276+0000","last_unstale":"2024-04-28T12:48:45.651078+0000","last_undegraded":"2024-04-28T12:48:45.651078+0000","last_fullsized":"2024-04-28T12:48:45.651078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:49.264624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.650486+0000","last_change":"2024-04-28T12:48:44.472196+0000","last_active":"2024-04-28T12:48:45.650486+0000","last_peered":"2024-04-28T12:48:45.650486+0000","last_clean":"2024-04-28T12:48:45.650486+0000","last_became_active":"2024-04-28T12:48:44.471857+0000","last_became_peered":"2024-04-28T12:48:44.471857+0000","last_unstale":"2024-04-28T12:48:45.650486+0000","last_undegraded":"2024-04-28T12:48:45.650486+0000","last_fullsized":"2024-04-28T12:48:45.650486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:43.813225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-28T12:48:45.735775+0000","last_change":"2024-04-28T12:48:44.478788+0000","last_active":"2024-04-28T12:48:45.735775+0000","last_peered":"2024-04-28T12:48:45.735775+0000","last_clean":"2024-04-28T12:48:45.735775+0000","last_became_active":"2024-04-28T12:48:44.478449+0000","last_became_peered":"2024-04-28T12:48:44.478449+0000","last_unstale":"2024-04-28T12:48:45.735775+0000","last_undegraded":"2024-04-28T12:48:45.735775+0000","last_fullsized":"2024-04-28T12:48:45.735775+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:48:11.638628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283884,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":125,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":34949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1598,"internal_metadata":36174274},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299428,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":1033849,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1587,"internal_metadata":31914445},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":441779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":449784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":443050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":444534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":441856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888115,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":442216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724589,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":1033416,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:48:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-28T12:48:52.643 INFO:tasks.cephadm:Setting up client nodes... 2024-04-28T12:48:52.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-28T12:48:52.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:52 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1689149449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:52.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:52 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3158531469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:52.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:52 smithi053 ceph-mon[29182]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:48:52.923 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:48:52.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:52 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1689149449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:52.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:52 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3158531469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:52 smithi077 ceph-mon[32553]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:48:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:52 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1689149449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:52 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3158531469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:53.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:52 smithi191 ceph-mon[33016]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:48:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:53 smithi053 ceph-mon[29182]: pgmap v295: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.9 KiB/s wr, 17 op/s 2024-04-28T12:48:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:53 smithi053 ceph-mon[29182]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:48:53.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:53 smithi077 ceph-mon[32553]: pgmap v295: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.9 KiB/s wr, 17 op/s 2024-04-28T12:48:53.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:53 smithi077 ceph-mon[32553]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:48:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:53 smithi191 ceph-mon[33016]: pgmap v295: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.9 KiB/s wr, 17 op/s 2024-04-28T12:48:54.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:53 smithi191 ceph-mon[33016]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:48:54.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 -- 172.21.15.53:0/1379255464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 msgr2=0x7f933010b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 --2- 172.21.15.53:0/1379255464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f933010b920 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f932800b0d0 tx=0x7f932800b3e0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 -- 172.21.15.53:0/1379255464 shutdown_connections 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 --2- 172.21.15.53:0/1379255464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f933010b920 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 --2- 172.21.15.53:0/1379255464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93301034d0 0x7f93301038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 --2- 172.21.15.53:0/1379255464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93301046d0 0x7f9330104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 -- 172.21.15.53:0/1379255464 >> 172.21.15.53:0/1379255464 conn(0x7f93300fea70 msgr2=0x7f9330100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 -- 172.21.15.53:0/1379255464 shutdown_connections 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.584+0000 7f93371ed700 1 -- 172.21.15.53:0/1379255464 wait complete. 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 Processor -- start 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 -- start start 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93301034d0 0x7f9330199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93301046d0 0x7f9330199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f93301a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f933010ef50 con 0x7f9330105070 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f933010edd0 con 0x7f93301034d0 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f93371ed700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f933010f0d0 con 0x7f93301046d0 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f93301a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f93301a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38714/0 (socket says 172.21.15.53:38714) 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 -- 172.21.15.53:0/3096909508 learned_addr learned my addr 172.21.15.53:0/3096909508 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f932ffff700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93301046d0 0x7f9330199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f9334f89700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93301034d0 0x7f9330199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 -- 172.21.15.53:0/3096909508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93301046d0 msgr2=0x7f9330199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93301046d0 0x7f9330199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 -- 172.21.15.53:0/3096909508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93301034d0 msgr2=0x7f9330199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93301034d0 0x7f9330199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.585+0000 7f933578a700 1 -- 172.21.15.53:0/3096909508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93301a0fa0 con 0x7f9330105070 2024-04-28T12:48:54.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.586+0000 7f933578a700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f93301a0900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f932800d010 tx=0x7f9328004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:54.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.586+0000 7f932dffb700 1 -- 172.21.15.53:0/3096909508 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932801f030 con 0x7f9330105070 2024-04-28T12:48:54.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.586+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9330075650 con 0x7f9330105070 2024-04-28T12:48:54.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.586+0000 7f932dffb700 1 -- 172.21.15.53:0/3096909508 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9328003a00 con 0x7f9330105070 2024-04-28T12:48:54.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.586+0000 7f932dffb700 1 -- 172.21.15.53:0/3096909508 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328004740 con 0x7f9330105070 2024-04-28T12:48:54.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.586+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9330075b80 con 0x7f9330105070 2024-04-28T12:48:54.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.587+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9330051960 con 0x7f9330105070 2024-04-28T12:48:54.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.588+0000 7f932dffb700 1 -- 172.21.15.53:0/3096909508 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9328007a00 con 0x7f9330105070 2024-04-28T12:48:54.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.589+0000 7f932dffb700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f93180a3d80 0x7f93180a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.589+0000 7f9334f89700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f93180a3d80 0x7f93180a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.589+0000 7f932dffb700 1 -- 172.21.15.53:0/3096909508 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f932800f020 con 0x7f9330105070 2024-04-28T12:48:54.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.589+0000 7f9334f89700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f93180a3d80 0x7f93180a6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9330104530 tx=0x7f932000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:54.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.593+0000 7f932dffb700 1 -- 172.21.15.53:0/3096909508 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9328029020 con 0x7f9330105070 2024-04-28T12:48:54.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.887+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9330076f60 con 0x7f9330105070 2024-04-28T12:48:54.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.889+0000 7f932dffb700 1 -- 172.21.15.53:0/3096909508 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f932809e590 con 0x7f9330105070 2024-04-28T12:48:54.885 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-04-28T12:48:54.885 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC2RS5mtED7NBAAV84eprpc1GzqDZ4TisXiOg== 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f93180a3d80 msgr2=0x7f93180a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f93180a3d80 0x7f93180a6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9330104530 tx=0x7f932000a3b0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 msgr2=0x7f93301a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f93301a0900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f932800d010 tx=0x7f9328004e40 comp rx=0 tx=0).stop 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 shutdown_connections 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f93180a3d80 0x7f93180a6240 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9330105070 0x7f93301a0900 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93301034d0 0x7f9330199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 --2- 172.21.15.53:0/3096909508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93301046d0 0x7f9330199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 >> 172.21.15.53:0/3096909508 conn(0x7f93300fea70 msgr2=0x7f9330107900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 shutdown_connections 2024-04-28T12:48:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:54.892+0000 7f93371ed700 1 -- 172.21.15.53:0/3096909508 wait complete. 2024-04-28T12:48:55.531 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:48:55.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-28T12:48:55.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-28T12:48:55.570 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-28T12:48:55.571 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-28T12:48:55.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-28T12:48:55.856 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:48:55.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:55 smithi053 ceph-mon[29182]: pgmap v296: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.0 KiB/s wr, 17 op/s 2024-04-28T12:48:55.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:55 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3096909508' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T12:48:55.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:55 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3096909508' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T12:48:55.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:55 smithi077 ceph-mon[32553]: pgmap v296: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.0 KiB/s wr, 17 op/s 2024-04-28T12:48:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:55 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3096909508' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T12:48:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:55 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3096909508' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T12:48:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:55 smithi191 ceph-mon[33016]: pgmap v296: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 8.0 KiB/s wr, 17 op/s 2024-04-28T12:48:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:55 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3096909508' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T12:48:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:55 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3096909508' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T12:48:57.597 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:57 smithi053 ceph-mon[29182]: pgmap v297: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-04-28T12:48:57.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 -- 172.21.15.53:0/3566525741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c1094d0 msgr2=0x7f397c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 --2- 172.21.15.53:0/3566525741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c1094d0 0x7f397c1098b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3970009990 tx=0x7f3970009ca0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 -- 172.21.15.53:0/3566525741 shutdown_connections 2024-04-28T12:48:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 --2- 172.21.15.53:0/3566525741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c1094d0 0x7f397c1098b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 --2- 172.21.15.53:0/3566525741 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f397c06c590 0x7f397c110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 --2- 172.21.15.53:0/3566525741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c06bbd0 0x7f397c06c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 -- 172.21.15.53:0/3566525741 >> 172.21.15.53:0/3566525741 conn(0x7f397c0793c0 msgr2=0x7f397c0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 -- 172.21.15.53:0/3566525741 shutdown_connections 2024-04-28T12:48:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.854+0000 7f3982b94700 1 -- 172.21.15.53:0/3566525741 wait complete. 2024-04-28T12:48:57.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 Processor -- start 2024-04-28T12:48:57.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 -- start start 2024-04-28T12:48:57.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c06bbd0 0x7f397c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c06c590 0x7f397c19c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f397c1094d0 0x7f397c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f397c113660 con 0x7f397c06c590 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f397c1134e0 con 0x7f397c06bbd0 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f3982b94700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f397c1137e0 con 0x7f397c1094d0 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.855+0000 7f397bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c06c590 0x7f397c19c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c06c590 0x7f397c19c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45680/0 (socket says 172.21.15.53:45680) 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 -- 172.21.15.53:0/716241822 learned_addr learned my addr 172.21.15.53:0/716241822 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f3980930700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c06bbd0 0x7f397c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f3981131700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f397c1094d0 0x7f397c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 -- 172.21.15.53:0/716241822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f397c1094d0 msgr2=0x7f397c1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f397c1094d0 0x7f397c1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 -- 172.21.15.53:0/716241822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c06bbd0 msgr2=0x7f397c19be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c06bbd0 0x7f397c19be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 -- 172.21.15.53:0/716241822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397c1a0dd0 con 0x7f397c06c590 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f3981131700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f397c1094d0 0x7f397c1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:48:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f397bfff700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c06c590 0x7f397c19c350 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f396800b530 tx=0x7f396800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:57.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f3979ffb700 1 -- 172.21.15.53:0/716241822 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968015020 con 0x7f397c06c590 2024-04-28T12:48:57.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f3979ffb700 1 -- 172.21.15.53:0/716241822 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3968011ce0 con 0x7f397c06c590 2024-04-28T12:48:57.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f3979ffb700 1 -- 172.21.15.53:0/716241822 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968010e40 con 0x7f397c06c590 2024-04-28T12:48:57.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.856+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397c1a10c0 con 0x7f397c06c590 2024-04-28T12:48:57.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.857+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f397c1a8940 con 0x7f397c06c590 2024-04-28T12:48:57.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.858+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f397c06a110 con 0x7f397c06c590 2024-04-28T12:48:57.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.859+0000 7f3979ffb700 1 -- 172.21.15.53:0/716241822 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3968018020 con 0x7f397c06c590 2024-04-28T12:48:57.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.859+0000 7f3979ffb700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f396c0a3d50 0x7f396c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.860+0000 7f3980930700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f396c0a3d50 0x7f396c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.860+0000 7f3979ffb700 1 -- 172.21.15.53:0/716241822 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f39680cfac0 con 0x7f397c06c590 2024-04-28T12:48:57.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.860+0000 7f3980930700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f396c0a3d50 0x7f396c0a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f39700098e0 tx=0x7f3970005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:57.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:57.865+0000 7f3979ffb700 1 -- 172.21.15.53:0/716241822 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f396801f020 con 0x7f397c06c590 2024-04-28T12:48:57.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:57 smithi077 ceph-mon[32553]: pgmap v297: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-04-28T12:48:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:57 smithi191 ceph-mon[33016]: pgmap v297: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-04-28T12:48:58.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.133+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f397c02e490 con 0x7f397c06c590 2024-04-28T12:48:58.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.138+0000 7f3979ffb700 1 -- 172.21.15.53:0/716241822 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264998 (secure 0 0 0) 0x7f396809c600 con 0x7f397c06c590 2024-04-28T12:48:58.138 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:48:58.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f396c0a3d50 msgr2=0x7f396c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f396c0a3d50 0x7f396c0a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f39700098e0 tx=0x7f3970005960 comp rx=0 tx=0).stop 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c06c590 msgr2=0x7f397c19c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c06c590 0x7f397c19c350 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f396800b530 tx=0x7f396800b840 comp rx=0 tx=0).stop 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 shutdown_connections 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f396c0a3d50 0x7f396c0a6210 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f397c1094d0 0x7f397c1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c06bbd0 0x7f397c19be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 --2- 172.21.15.53:0/716241822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f397c06c590 0x7f397c19c350 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 >> 172.21.15.53:0/716241822 conn(0x7f397c0793c0 msgr2=0x7f397c10e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 shutdown_connections 2024-04-28T12:48:58.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:48:58.144+0000 7f3982b94700 1 -- 172.21.15.53:0/716241822 wait complete. 2024-04-28T12:48:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:58 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/716241822' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T12:48:58.884 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:58 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/716241822' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T12:48:58.917 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":3118663054},{"type":"v1","addr":"172.21.15.53:6801","nonce":3118663054}]},"active_addr":"172.21.15.53:6801/3118663054","active_change":"2024-04-28T12:41:34.367987+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.53:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1416593328}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":2977555317}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1515104761}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":2646682174}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1570691245}]}]} 2024-04-28T12:48:58.924 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-28T12:48:58.924 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-28T12:48:58.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T12:48:58.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:58 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/716241822' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T12:48:59.178 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:48:59.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:48:59 smithi053 ceph-mon[29182]: pgmap v298: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.1 KiB/s wr, 15 op/s 2024-04-28T12:48:59.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:48:59 smithi077 ceph-mon[32553]: pgmap v298: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.1 KiB/s wr, 15 op/s 2024-04-28T12:49:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:48:59 smithi191 ceph-mon[33016]: pgmap v298: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.1 KiB/s wr, 15 op/s 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.937+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/718549245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78103730 msgr2=0x7f7d78106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.937+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/718549245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78103730 0x7f7d78106fc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d600099c0 tx=0x7f7d60009cd0 comp rx=0 tx=0).stop 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/718549245 shutdown_connections 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/718549245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d781076a0 0x7f7d7810f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/718549245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d78102d80 0x7f7d78103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/718549245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78103730 0x7f7d78106fc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/718549245 >> 172.21.15.53:0/718549245 conn(0x7f7d78078990 msgr2=0x7f7d780ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/718549245 shutdown_connections 2024-04-28T12:49:00.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/718549245 wait complete. 2024-04-28T12:49:00.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.938+0000 7f7d7ea3d700 1 Processor -- start 2024-04-28T12:49:00.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7ea3d700 1 -- start start 2024-04-28T12:49:00.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7ea3d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78102d80 0x7f7d7819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:00.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7ea3d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d78103730 0x7f7d7819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:00.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7ea3d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d781076a0 0x7f7d78198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:00.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7ea3d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781125c0 con 0x7f7d78102d80 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7ea3d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78112440 con 0x7f7d781076a0 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7ea3d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78112740 con 0x7f7d78103730 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78102d80 0x7f7d7819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78102d80 0x7f7d7819e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45698/0 (socket says 172.21.15.53:45698) 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 -- 172.21.15.53:0/531697486 learned_addr learned my addr 172.21.15.53:0/531697486 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d7cfda700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d781076a0 0x7f7d78198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d6ffff700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d78103730 0x7f7d7819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 -- 172.21.15.53:0/531697486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d78103730 msgr2=0x7f7d7819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d78103730 0x7f7d7819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 -- 172.21.15.53:0/531697486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d781076a0 msgr2=0x7f7d78198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d781076a0 0x7f7d78198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 -- 172.21.15.53:0/531697486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d781990e0 con 0x7f7d78102d80 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.939+0000 7f7d77fff700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78102d80 0x7f7d7819e720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800d950 tx=0x7f7d6800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:00.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.940+0000 7f7d75ffb700 1 -- 172.21.15.53:0/531697486 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68016070 con 0x7f7d78102d80 2024-04-28T12:49:00.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.940+0000 7f7d75ffb700 1 -- 172.21.15.53:0/531697486 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d6800fd40 con 0x7f7d78102d80 2024-04-28T12:49:00.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.940+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d781993d0 con 0x7f7d78102d80 2024-04-28T12:49:00.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.940+0000 7f7d75ffb700 1 -- 172.21.15.53:0/531697486 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800e3b0 con 0x7f7d78102d80 2024-04-28T12:49:00.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.940+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d781a55f0 con 0x7f7d78102d80 2024-04-28T12:49:00.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.941+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7806a110 con 0x7f7d78102d80 2024-04-28T12:49:00.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.942+0000 7f7d75ffb700 1 -- 172.21.15.53:0/531697486 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7d68019070 con 0x7f7d78102d80 2024-04-28T12:49:00.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.943+0000 7f7d75ffb700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d640a3d50 0x7f7d640a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:00.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.943+0000 7f7d6ffff700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d640a3d50 0x7f7d640a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:00.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.943+0000 7f7d75ffb700 1 -- 172.21.15.53:0/531697486 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7d680cfe10 con 0x7f7d78102d80 2024-04-28T12:49:00.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.943+0000 7f7d6ffff700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d640a3d50 0x7f7d640a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d60006010 tx=0x7f7d60005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:00.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:00.947+0000 7f7d75ffb700 1 -- 172.21.15.53:0/531697486 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7d6809c840 con 0x7f7d78102d80 2024-04-28T12:49:01.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.155+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d78103b10 con 0x7f7d78102d80 2024-04-28T12:49:01.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.157+0000 7f7d75ffb700 1 -- 172.21.15.53:0/531697486 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22184 (secure 0 0 0) 0x7f7d6809c140 con 0x7f7d78102d80 2024-04-28T12:49:01.153 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:49:01.153 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":86,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:48:45.456811+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d640a3d50 msgr2=0x7f7d640a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d640a3d50 0x7f7d640a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d60006010 tx=0x7f7d60005a90 comp rx=0 tx=0).stop 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78102d80 msgr2=0x7f7d7819e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78102d80 0x7f7d7819e720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800d950 tx=0x7f7d6800dc60 comp rx=0 tx=0).stop 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 shutdown_connections 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7d640a3d50 0x7f7d640a6210 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d78102d80 0x7f7d7819e720 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:01.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d781076a0 0x7f7d78198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:01.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 --2- 172.21.15.53:0/531697486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d78103730 0x7f7d7819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:01.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 >> 172.21.15.53:0/531697486 conn(0x7f7d78078990 msgr2=0x7f7d780ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:01.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 shutdown_connections 2024-04-28T12:49:01.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:01.159+0000 7f7d7ea3d700 1 -- 172.21.15.53:0/531697486 wait complete. 2024-04-28T12:49:01.784 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:01 smithi053 ceph-mon[29182]: pgmap v299: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-04-28T12:49:01.784 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:01 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/531697486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:01.821 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-28T12:49:01.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T12:49:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:01 smithi077 ceph-mon[32553]: pgmap v299: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-04-28T12:49:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:01 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/531697486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:01 smithi191 ceph-mon[33016]: pgmap v299: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-04-28T12:49:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:01 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/531697486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:02.076 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:03.614 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:03 smithi053 ceph-mon[29182]: pgmap v300: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-28T12:49:03.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 -- 172.21.15.53:0/3062233828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103470 msgr2=0x7f95081038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:03.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 --2- 172.21.15.53:0/3062233828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103470 0x7f95081038f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009990 tx=0x7f94f8009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:03.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 -- 172.21.15.53:0/3062233828 shutdown_connections 2024-04-28T12:49:03.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 --2- 172.21.15.53:0/3062233828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9508103e30 0x7f9508110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:03.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 --2- 172.21.15.53:0/3062233828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95081094c0 0x7f95081098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:03.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 --2- 172.21.15.53:0/3062233828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103470 0x7f95081038f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 -- 172.21.15.53:0/3062233828 >> 172.21.15.53:0/3062233828 conn(0x7f950807a580 msgr2=0x7f950807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 -- 172.21.15.53:0/3062233828 shutdown_connections 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.874+0000 7f95103cd700 1 -- 172.21.15.53:0/3062233828 wait complete. 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 Processor -- start 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 -- start start 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9508103470 0x7f950819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103e30 0x7f950819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95081094c0 0x7f95081998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f95081135c0 con 0x7f9508103e30 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9508113440 con 0x7f95081094c0 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f95103cd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9508113740 con 0x7f9508103470 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f950d968700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103e30 0x7f950819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f950d968700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103e30 0x7f950819fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45714/0 (socket says 172.21.15.53:45714) 2024-04-28T12:49:03.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.875+0000 7f950d968700 1 -- 172.21.15.53:0/4242684650 learned_addr learned my addr 172.21.15.53:0/4242684650 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950e169700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9508103470 0x7f950819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950e96a700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95081094c0 0x7f95081998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950d968700 1 -- 172.21.15.53:0/4242684650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9508103470 msgr2=0x7f950819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950d968700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9508103470 0x7f950819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950d968700 1 -- 172.21.15.53:0/4242684650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95081094c0 msgr2=0x7f95081998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950d968700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95081094c0 0x7f95081998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950d968700 1 -- 172.21.15.53:0/4242684650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950819a000 con 0x7f9508103e30 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950e169700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9508103470 0x7f950819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f950d968700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103e30 0x7f950819fc20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f94f8000c00 tx=0x7f94f80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f94ff7fe700 1 -- 172.21.15.53:0/4242684650 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8027070 con 0x7f9508103e30 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f94ff7fe700 1 -- 172.21.15.53:0/4242684650 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94f80041d0 con 0x7f9508103e30 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950819a290 con 0x7f9508103e30 2024-04-28T12:49:03.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f94ff7fe700 1 -- 172.21.15.53:0/4242684650 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f800f460 con 0x7f9508103e30 2024-04-28T12:49:03.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.876+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a65b0 con 0x7f9508103e30 2024-04-28T12:49:03.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.877+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f950806a110 con 0x7f9508103e30 2024-04-28T12:49:03.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.879+0000 7f94ff7fe700 1 -- 172.21.15.53:0/4242684650 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f94f802a070 con 0x7f9508103e30 2024-04-28T12:49:03.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.880+0000 7f94ff7fe700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f94f40a3cd0 0x7f94f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:03.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.880+0000 7f950e169700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f94f40a3cd0 0x7f94f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:03.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.880+0000 7f94ff7fe700 1 -- 172.21.15.53:0/4242684650 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f94f80d17c0 con 0x7f9508103e30 2024-04-28T12:49:03.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.880+0000 7f950e169700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f94f40a3cd0 0x7f94f40a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9504005ea0 tx=0x7f9504005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:03.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:03.884+0000 7f94ff7fe700 1 -- 172.21.15.53:0/4242684650 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f94f8031070 con 0x7f9508103e30 2024-04-28T12:49:03.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:03 smithi077 ceph-mon[32553]: pgmap v300: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-28T12:49:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:03 smithi191 ceph-mon[33016]: pgmap v300: 97 pgs: 97 active+clean; 624 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-28T12:49:04.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.092+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f950802e490 con 0x7f9508103e30 2024-04-28T12:49:04.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.093+0000 7f94ff7fe700 1 -- 172.21.15.53:0/4242684650 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22184 (secure 0 0 0) 0x7f94f8013070 con 0x7f9508103e30 2024-04-28T12:49:04.089 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:49:04.089 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":86,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:48:45.456811+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:49:04.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f94f40a3cd0 msgr2=0x7f94f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f94f40a3cd0 0x7f94f40a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9504005ea0 tx=0x7f9504005e30 comp rx=0 tx=0).stop 2024-04-28T12:49:04.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103e30 msgr2=0x7f950819fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103e30 0x7f950819fc20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f94f8000c00 tx=0x7f94f80047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 shutdown_connections 2024-04-28T12:49:04.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f94f40a3cd0 0x7f94f40a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95081094c0 0x7f95081998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9508103470 0x7f950819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 --2- 172.21.15.53:0/4242684650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9508103e30 0x7f950819fc20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 >> 172.21.15.53:0/4242684650 conn(0x7f950807a580 msgr2=0x7f9508102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:04.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 shutdown_connections 2024-04-28T12:49:04.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:04.095+0000 7f95103cd700 1 -- 172.21.15.53:0/4242684650 wait complete. 2024-04-28T12:49:04.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:04 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4242684650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:04.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-28T12:49:04.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-28T12:49:04.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-28T12:49:04.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-28T12:49:04.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-28T12:49:04.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-28T12:49:04.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-28T12:49:04.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-28T12:49:04.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-28T12:49:04.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-28T12:49:04.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-28T12:49:04.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-28T12:49:04.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:04 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4242684650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:04 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4242684650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:05.793 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:05.812 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:05.818 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:05.822 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:05.822 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:05.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:05 smithi053 ceph-mon[29182]: pgmap v301: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.8 KiB/s wr, 9 op/s 2024-04-28T12:49:05.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:05 smithi077 ceph-mon[32553]: pgmap v301: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.8 KiB/s wr, 9 op/s 2024-04-28T12:49:06.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:05 smithi191 ceph-mon[33016]: pgmap v301: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.8 KiB/s wr, 9 op/s 2024-04-28T12:49:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:06.852 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:06.852 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:06.852 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:07.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:07 smithi053 ceph-mon[29182]: pgmap v302: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-28T12:49:07.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:07 smithi077 ceph-mon[32553]: pgmap v302: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-28T12:49:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:07 smithi191 ceph-mon[33016]: pgmap v302: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-28T12:49:09.617 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:09 smithi053 ceph-mon[29182]: pgmap v303: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-28T12:49:09.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 -- 172.21.15.53:0/409649858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c1094b0 msgr2=0x7f0e8c109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 --2- 172.21.15.53:0/409649858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c1094b0 0x7f0e8c109890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c009990 tx=0x7f0e7c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 -- 172.21.15.53:0/409649858 shutdown_connections 2024-04-28T12:49:09.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 --2- 172.21.15.53:0/409649858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c1094b0 0x7f0e8c109890 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 --2- 172.21.15.53:0/409649858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c1023b0 0x7f0e8c1102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 --2- 172.21.15.53:0/409649858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e8c101a10 0x7f0e8c101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 -- 172.21.15.53:0/409649858 >> 172.21.15.53:0/409649858 conn(0x7f0e8c078990 msgr2=0x7f0e8c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:09.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 -- 172.21.15.53:0/409649858 shutdown_connections 2024-04-28T12:49:09.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.840+0000 7f0e9360d700 1 -- 172.21.15.53:0/409649858 wait complete. 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 Processor -- start 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 -- start start 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c101a10 0x7f0e8c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c1023b0 0x7f0e8c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e8c1094b0 0x7f0e8c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c113620 con 0x7f0e8c101a10 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1134a0 con 0x7f0e8c1094b0 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e9360d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1137a0 con 0x7f0e8c1023b0 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e913a9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c101a10 0x7f0e8c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e913a9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c101a10 0x7f0e8c19bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36986/0 (socket says 172.21.15.53:36986) 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.841+0000 7f0e913a9700 1 -- 172.21.15.53:0/1490796225 learned_addr learned my addr 172.21.15.53:0/1490796225 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e91baa700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e8c1094b0 0x7f0e8c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e90ba8700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c1023b0 0x7f0e8c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e913a9700 1 -- 172.21.15.53:0/1490796225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c1023b0 msgr2=0x7f0e8c19c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e913a9700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c1023b0 0x7f0e8c19c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e913a9700 1 -- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e8c1094b0 msgr2=0x7f0e8c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e913a9700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e8c1094b0 0x7f0e8c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e913a9700 1 -- 172.21.15.53:0/1490796225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c1a0da0 con 0x7f0e8c101a10 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e90ba8700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c1023b0 0x7f0e8c19c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e913a9700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c101a10 0x7f0e8c19bd90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c000c00 tx=0x7f0e7c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c027070 con 0x7f0e8c101a10 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c1a1030 con 0x7f0e8c101a10 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e7c003f20 con 0x7f0e8c101a10 2024-04-28T12:49:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c022cf0 con 0x7f0e8c101a10 2024-04-28T12:49:09.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.842+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8c1a8910 con 0x7f0e8c101a10 2024-04-28T12:49:09.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.843+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e8c063bb0 con 0x7f0e8c101a10 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.844+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0e7c02a070 con 0x7f0e8c101a10 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.845+0000 7f0e827fc700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0e780a3d50 0x7f0e780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.845+0000 7f0e90ba8700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0e780a3d50 0x7f0e780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.845+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0e7c0d1880 con 0x7f0e8c101a10 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.845+0000 7f0e827fc700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] conn(0x7f0e780ad530 0x7f0e780af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.845+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 --> [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e7c0056d0 con 0x7f0e780ad530 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.845+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f0e7c0d1c30 con 0x7f0e8c101a10 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.845+0000 7f0e90ba8700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0e780a3d50 0x7f0e780a6210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e88005950 tx=0x7f0e880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:09.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.846+0000 7f0e91baa700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] conn(0x7f0e780ad530 0x7f0e780af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.846+0000 7f0e91baa700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] conn(0x7f0e780ad530 0x7f0e780af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:09.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.847+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== osd.1 v2:172.21.15.77:6800/2126226610 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e7c0056d0 con 0x7f0e780ad530 2024-04-28T12:49:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.864+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 --> [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e8c063fb0 con 0x7f0e780ad530 2024-04-28T12:49:09.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.864+0000 7f0e827fc700 1 -- 172.21.15.53:0/1490796225 <== osd.1 v2:172.21.15.77:6800/2126226610 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0e8c063fb0 con 0x7f0e780ad530 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] conn(0x7f0e780ad530 msgr2=0x7f0e780af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] conn(0x7f0e780ad530 0x7f0e780af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0e780a3d50 msgr2=0x7f0e780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0e780a3d50 0x7f0e780a6210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e88005950 tx=0x7f0e880058e0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c101a10 msgr2=0x7f0e8c19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 --2- 172.21.15.53:0/1490796225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e8c101a10 0x7f0e8c19bd90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c000c00 tx=0x7f0e7c0047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e91baa700 1 -- 172.21.15.53:0/1490796225 reap_dead start 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 shutdown_connections 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 >> 172.21.15.53:0/1490796225 conn(0x7f0e8c078990 msgr2=0x7f0e8c106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 shutdown_connections 2024-04-28T12:49:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:09.865+0000 7f0e9360d700 1 -- 172.21.15.53:0/1490796225 wait complete. 2024-04-28T12:49:09.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:09 smithi077 ceph-mon[32553]: pgmap v303: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-28T12:49:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:09 smithi191 ceph-mon[33016]: pgmap v303: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-28T12:49:10.917 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:10.918 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:11.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:11 smithi053 ceph-mon[29182]: pgmap v304: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-28T12:49:11.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:11 smithi077 ceph-mon[32553]: pgmap v304: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-28T12:49:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:11 smithi191 ceph-mon[33016]: pgmap v304: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-28T12:49:13.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.446+0000 7f63b8760700 1 -- 172.21.15.53:0/1595158527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b01094a0 msgr2=0x7f63b0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.446+0000 7f63b8760700 1 --2- 172.21.15.53:0/1595158527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b01094a0 0x7f63b0109880 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009990 tx=0x7f63ac009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.446+0000 7f63b8760700 1 -- 172.21.15.53:0/1595158527 shutdown_connections 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.446+0000 7f63b8760700 1 --2- 172.21.15.53:0/1595158527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b01094a0 0x7f63b0109880 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.446+0000 7f63b8760700 1 --2- 172.21.15.53:0/1595158527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b006bad0 0x7f63b01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.446+0000 7f63b8760700 1 --2- 172.21.15.53:0/1595158527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b006b130 0x7f63b006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 -- 172.21.15.53:0/1595158527 >> 172.21.15.53:0/1595158527 conn(0x7f63b00792d0 msgr2=0x7f63b00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 -- 172.21.15.53:0/1595158527 shutdown_connections 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 -- 172.21.15.53:0/1595158527 wait complete. 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 Processor -- start 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 -- start start 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b006b130 0x7f63b01023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b006bad0 0x7f63b0102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:13.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.447+0000 7f63b8760700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b01094a0 0x7f63b0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b8760700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01135b0 con 0x7f63b006bad0 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b8760700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0113430 con 0x7f63b006b130 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b8760700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0113730 con 0x7f63b01094a0 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b64fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b006b130 0x7f63b01023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b5cfb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b006bad0 0x7f63b0102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b64fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b006b130 0x7f63b01023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.53:40436/0 (socket says 172.21.15.53:40436) 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b64fc700 1 -- 172.21.15.53:0/407885295 learned_addr learned my addr 172.21.15.53:0/407885295 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b6cfd700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b01094a0 0x7f63b0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b5cfb700 1 -- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b01094a0 msgr2=0x7f63b0102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b5cfb700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63b01094a0 0x7f63b0102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b5cfb700 1 -- 172.21.15.53:0/407885295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b006b130 msgr2=0x7f63b01023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b5cfb700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b006b130 0x7f63b01023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b5cfb700 1 -- 172.21.15.53:0/407885295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b01a8090 con 0x7f63b006bad0 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b5cfb700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b006bad0 0x7f63b0102930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f63a000d750 tx=0x7f63a000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:13.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0017070 con 0x7f63b006bad0 2024-04-28T12:49:13.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a0005020 con 0x7f63b006bad0 2024-04-28T12:49:13.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b01a8380 con 0x7f63b006bad0 2024-04-28T12:49:13.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a000fa10 con 0x7f63b006bad0 2024-04-28T12:49:13.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.448+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b01a8870 con 0x7f63b006bad0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.449+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f63b0063b30 con 0x7f63b006bad0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.451+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f63a000fbd0 con 0x7f63b006bad0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.451+0000 7f63a77fe700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f639c0a3cd0 0x7f639c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.451+0000 7f63b64fc700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f639c0a3cd0 0x7f639c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.451+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f63a00cffe0 con 0x7f63b006bad0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.452+0000 7f63a77fe700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] conn(0x7f639c0ad4b0 0x7f639c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.452+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 --> [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63a000fec0 con 0x7f639c0ad4b0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.452+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f63a00d04c0 con 0x7f63b006bad0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.452+0000 7f63b64fc700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f639c0a3cd0 0x7f639c0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f63ac003800 tx=0x7f63ac005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:13.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.452+0000 7f63b6cfd700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] conn(0x7f639c0ad4b0 0x7f639c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:13.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.452+0000 7f63b6cfd700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] conn(0x7f639c0ad4b0 0x7f639c0af8d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:13.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.453+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== osd.11 v2:172.21.15.191:6824/1302539827 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63a000fec0 con 0x7f639c0ad4b0 2024-04-28T12:49:13.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.471+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 --> [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f63b0063fb0 con 0x7f639c0ad4b0 2024-04-28T12:49:13.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.471+0000 7f63a77fe700 1 -- 172.21.15.53:0/407885295 <== osd.11 v2:172.21.15.191:6824/1302539827 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f63b0063fb0 con 0x7f639c0ad4b0 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] conn(0x7f639c0ad4b0 msgr2=0x7f639c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] conn(0x7f639c0ad4b0 0x7f639c0af8d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f639c0a3cd0 msgr2=0x7f639c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f639c0a3cd0 0x7f639c0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f63ac003800 tx=0x7f63ac005be0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b006bad0 msgr2=0x7f63b0102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 --2- 172.21.15.53:0/407885295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63b006bad0 0x7f63b0102930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f63a000d750 tx=0x7f63a000da60 comp rx=0 tx=0).stop 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b6cfd700 1 -- 172.21.15.53:0/407885295 reap_dead start 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 shutdown_connections 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 >> 172.21.15.53:0/407885295 conn(0x7f63b00792d0 msgr2=0x7f63b01004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 shutdown_connections 2024-04-28T12:49:13.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:13.472+0000 7f63b8760700 1 -- 172.21.15.53:0/407885295 wait complete. 2024-04-28T12:49:13.623 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:13 smithi053 ceph-mon[29182]: pgmap v305: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:49:13.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:13 smithi077 ceph-mon[32553]: pgmap v305: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:49:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:13 smithi191 ceph-mon[33016]: pgmap v305: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:49:15.731 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:15 smithi053 ceph-mon[29182]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:49:15.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:15 smithi077 ceph-mon[32553]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:49:16.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:15 smithi191 ceph-mon[33016]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:49:16.964 INFO:teuthology.orchestra.run.smithi053.stdout:55834574917 2024-04-28T12:49:16.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-28T12:49:17.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:17 smithi053 ceph-mon[29182]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:17.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:17 smithi077 ceph-mon[32553]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:18.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:17 smithi191 ceph-mon[33016]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 -- 172.21.15.53:0/2082223943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406c580 msgr2=0x7fc6041102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 --2- 172.21.15.53:0/2082223943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406c580 0x7fc6041102b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00b0d0 tx=0x7fc5fc00b3e0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 -- 172.21.15.53:0/2082223943 shutdown_connections 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 --2- 172.21.15.53:0/2082223943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406c580 0x7fc6041102b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 --2- 172.21.15.53:0/2082223943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc604109490 0x7fc604109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 --2- 172.21.15.53:0/2082223943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc60406bbc0 0x7fc60406c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 -- 172.21.15.53:0/2082223943 >> 172.21.15.53:0/2082223943 conn(0x7fc604078390 msgr2=0x7fc6040787a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 -- 172.21.15.53:0/2082223943 shutdown_connections 2024-04-28T12:49:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 -- 172.21.15.53:0/2082223943 wait complete. 2024-04-28T12:49:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 Processor -- start 2024-04-28T12:49:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.206+0000 7fc60afa5700 1 -- start start 2024-04-28T12:49:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc60afa5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406bbc0 0x7fc60419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc60afa5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc60406c580 0x7fc60419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc60afa5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc604109490 0x7fc6041998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc60afa5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc6041135f0 con 0x7fc60406bbc0 2024-04-28T12:49:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc60afa5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc604113470 con 0x7fc60406c580 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc60afa5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc604113770 con 0x7fc604109490 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406bbc0 0x7fc60419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406bbc0 0x7fc60419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48162/0 (socket says 172.21.15.53:48162) 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 -- 172.21.15.53:0/2240878250 learned_addr learned my addr 172.21.15.53:0/2240878250 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc603fff700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc60406c580 0x7fc60419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc609542700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc604109490 0x7fc6041998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 -- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc604109490 msgr2=0x7fc6041998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc604109490 0x7fc6041998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 -- 172.21.15.53:0/2240878250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc60406c580 msgr2=0x7fc60419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc60406c580 0x7fc60419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 -- 172.21.15.53:0/2240878250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60419a050 con 0x7fc60406bbc0 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc608d41700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406bbc0 0x7fc60419f6e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800d980 tx=0x7fc5f800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.207+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8016070 con 0x7fc60406bbc0 2024-04-28T12:49:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.208+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60419a340 con 0x7fc60406bbc0 2024-04-28T12:49:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.208+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5f80044d0 con 0x7fc60406bbc0 2024-04-28T12:49:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.208+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8004d00 con 0x7fc60406bbc0 2024-04-28T12:49:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.208+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6041a65b0 con 0x7fc60406bbc0 2024-04-28T12:49:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.209+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc604063b70 con 0x7fc60406bbc0 2024-04-28T12:49:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.210+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc5f8004020 con 0x7fc60406bbc0 2024-04-28T12:49:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.210+0000 7fc601ffb700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc5f40a3cd0 0x7fc5f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.210+0000 7fc603fff700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc5f40a3cd0 0x7fc5f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.210+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc5f80d0900 con 0x7fc60406bbc0 2024-04-28T12:49:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.211+0000 7fc601ffb700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] conn(0x7fc5f40ad4b0 0x7fc5f40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.211+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 --> [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5f8015ec0 con 0x7fc5f40ad4b0 2024-04-28T12:49:18.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.211+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fc5f80d0de0 con 0x7fc60406bbc0 2024-04-28T12:49:18.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.211+0000 7fc603fff700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc5f40a3cd0 0x7fc5f40a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0005950 tx=0x7fc5f00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:18.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.211+0000 7fc609542700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] conn(0x7fc5f40ad4b0 0x7fc5f40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:18.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.211+0000 7fc609542700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] conn(0x7fc5f40ad4b0 0x7fc5f40af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:18.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.212+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== osd.2 v2:172.21.15.191:6800/246351477 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc5f8015ec0 con 0x7fc5f40ad4b0 2024-04-28T12:49:18.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.229+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 --> [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc604063ff0 con 0x7fc5f40ad4b0 2024-04-28T12:49:18.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.229+0000 7fc601ffb700 1 -- 172.21.15.53:0/2240878250 <== osd.2 v2:172.21.15.191:6800/246351477 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc604063ff0 con 0x7fc5f40ad4b0 2024-04-28T12:49:18.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] conn(0x7fc5f40ad4b0 msgr2=0x7fc5f40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:18.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] conn(0x7fc5f40ad4b0 0x7fc5f40af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc5f40a3cd0 msgr2=0x7fc5f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:18.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc5f40a3cd0 0x7fc5f40a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0005950 tx=0x7fc5f00058e0 comp rx=0 tx=0).stop 2024-04-28T12:49:18.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406bbc0 msgr2=0x7fc60419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:18.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 --2- 172.21.15.53:0/2240878250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc60406bbc0 0x7fc60419f6e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800d980 tx=0x7fc5f800dd40 comp rx=0 tx=0).stop 2024-04-28T12:49:18.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc609542700 1 -- 172.21.15.53:0/2240878250 reap_dead start 2024-04-28T12:49:18.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 shutdown_connections 2024-04-28T12:49:18.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 >> 172.21.15.53:0/2240878250 conn(0x7fc604078390 msgr2=0x7fc6041010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:18.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 shutdown_connections 2024-04-28T12:49:18.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:18.230+0000 7fc60afa5700 1 -- 172.21.15.53:0/2240878250 wait complete. 2024-04-28T12:49:18.890 INFO:teuthology.orchestra.run.smithi053.stdout:347892350995 2024-04-28T12:49:18.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 -- 172.21.15.53:0/1179292819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102cd0 msgr2=0x7fc52410f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 --2- 172.21.15.53:0/1179292819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102cd0 0x7fc52410f1c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00b0a0 tx=0x7fc51c00b3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 -- 172.21.15.53:0/1179292819 shutdown_connections 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 --2- 172.21.15.53:0/1179292819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102cd0 0x7fc52410f1c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 --2- 172.21.15.53:0/1179292819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc524108380 0x7fc524108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 --2- 172.21.15.53:0/1179292819 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc524102310 0x7fc524102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 -- 172.21.15.53:0/1179292819 >> 172.21.15.53:0/1179292819 conn(0x7fc524079290 msgr2=0x7fc5240796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:19.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 -- 172.21.15.53:0/1179292819 shutdown_connections 2024-04-28T12:49:19.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.073+0000 7fc529a6a700 1 -- 172.21.15.53:0/1179292819 wait complete. 2024-04-28T12:49:19.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 Processor -- start 2024-04-28T12:49:19.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 -- start start 2024-04-28T12:49:19.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102310 0x7fc52419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:19.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc524102cd0 0x7fc52419adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:19.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc524108380 0x7fc5241a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241124d0 con 0x7fc524102310 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc524112350 con 0x7fc524108380 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc529a6a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc524112650 con 0x7fc524102cd0 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.074+0000 7fc5237fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102310 0x7fc52419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102310 0x7fc52419a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48174/0 (socket says 172.21.15.53:48174) 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 -- 172.21.15.53:0/420838154 learned_addr learned my addr 172.21.15.53:0/420838154 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc522ffd700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc524102cd0 0x7fc52419adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc523fff700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc524108380 0x7fc5241a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 -- 172.21.15.53:0/420838154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc524102cd0 msgr2=0x7fc52419adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc524102cd0 0x7fc52419adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 -- 172.21.15.53:0/420838154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc524108380 msgr2=0x7fc5241a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc524108380 0x7fc5241a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 -- 172.21.15.53:0/420838154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5241a5e70 con 0x7fc524102310 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc5237fe700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102310 0x7fc52419a8b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc51400d980 tx=0x7fc51400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:19.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc514016070 con 0x7fc524102310 2024-04-28T12:49:19.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc524198700 con 0x7fc524102310 2024-04-28T12:49:19.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc514004830 con 0x7fc524102310 2024-04-28T12:49:19.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5140054c0 con 0x7fc524102310 2024-04-28T12:49:19.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.075+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc524198c90 con 0x7fc524102310 2024-04-28T12:49:19.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.077+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc524063af0 con 0x7fc524102310 2024-04-28T12:49:19.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.079+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc514004220 con 0x7fc524102310 2024-04-28T12:49:19.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc520ff9700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc50c0a3d50 0x7fc50c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:19.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc522ffd700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc50c0a3d50 0x7fc50c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:19.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc5140cffe0 con 0x7fc524102310 2024-04-28T12:49:19.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc520ff9700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] conn(0x7fc50c0ad530 0x7fc50c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:19.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 --> [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc51401ce70 con 0x7fc50c0ad530 2024-04-28T12:49:19.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fc5140d2600 con 0x7fc524102310 2024-04-28T12:49:19.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc523fff700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] conn(0x7fc50c0ad530 0x7fc50c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:19.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc522ffd700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc50c0a3d50 0x7fc50c0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc524079900 tx=0x7fc518009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:19.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.080+0000 7fc523fff700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] conn(0x7fc50c0ad530 0x7fc50c0af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:19.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.081+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== osd.6 v2:172.21.15.53:6818/512872966 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc51401ce70 con 0x7fc50c0ad530 2024-04-28T12:49:19.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.097+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 --> [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc524063f20 con 0x7fc50c0ad530 2024-04-28T12:49:19.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc520ff9700 1 -- 172.21.15.53:0/420838154 <== osd.6 v2:172.21.15.53:6818/512872966 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc524063f20 con 0x7fc50c0ad530 2024-04-28T12:49:19.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] conn(0x7fc50c0ad530 msgr2=0x7fc50c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:19.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] conn(0x7fc50c0ad530 0x7fc50c0af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc50c0a3d50 msgr2=0x7fc50c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc50c0a3d50 0x7fc50c0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc524079900 tx=0x7fc518009260 comp rx=0 tx=0).stop 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102310 msgr2=0x7fc52419a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 --2- 172.21.15.53:0/420838154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc524102310 0x7fc52419a8b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc51400d980 tx=0x7fc51400dc90 comp rx=0 tx=0).stop 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc523fff700 1 -- 172.21.15.53:0/420838154 reap_dead start 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 shutdown_connections 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 >> 172.21.15.53:0/420838154 conn(0x7fc524079290 msgr2=0x7fc524100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 shutdown_connections 2024-04-28T12:49:19.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:19.098+0000 7fc529a6a700 1 -- 172.21.15.53:0/420838154 wait complete. 2024-04-28T12:49:19.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:19 smithi053 ceph-mon[29182]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:19.741 INFO:teuthology.orchestra.run.smithi053.stdout:77309411394 2024-04-28T12:49:19.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-28T12:49:19.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:19 smithi077 ceph-mon[32553]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:19 smithi191 ceph-mon[33016]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 -- 172.21.15.53:0/914918642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102ce0 msgr2=0x7f261410f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 --2- 172.21.15.53:0/914918642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102ce0 0x7f261410f1d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f260c00b0a0 tx=0x7f260c00b3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 -- 172.21.15.53:0/914918642 shutdown_connections 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 --2- 172.21.15.53:0/914918642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102ce0 0x7f261410f1d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 --2- 172.21.15.53:0/914918642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2614108390 0x7f2614108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 --2- 172.21.15.53:0/914918642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2614102320 0x7f26141027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 -- 172.21.15.53:0/914918642 >> 172.21.15.53:0/914918642 conn(0x7f26140792d0 msgr2=0x7f26140796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 -- 172.21.15.53:0/914918642 shutdown_connections 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.282+0000 7f2619624700 1 -- 172.21.15.53:0/914918642 wait complete. 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 Processor -- start 2024-04-28T12:49:21.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 -- start start 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102320 0x7f261419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2614102ce0 0x7f261419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2614108390 0x7f2614199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2614112530 con 0x7f2614102320 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26141123b0 con 0x7f2614102ce0 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2619624700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f26141126b0 con 0x7f2614108390 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2612ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102320 0x7f261419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2612ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102320 0x7f261419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48190/0 (socket says 172.21.15.53:48190) 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f2612ffd700 1 -- 172.21.15.53:0/1057905525 learned_addr learned my addr 172.21.15.53:0/1057905525 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f26137fe700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2614108390 0x7f2614199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.283+0000 7f26127fc700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2614102ce0 0x7f261419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2612ffd700 1 -- 172.21.15.53:0/1057905525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2614108390 msgr2=0x7f2614199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2612ffd700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2614108390 0x7f2614199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2612ffd700 1 -- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2614102ce0 msgr2=0x7f261419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:21.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2612ffd700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2614102ce0 0x7f261419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:21.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2612ffd700 1 -- 172.21.15.53:0/1057905525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261419a0c0 con 0x7f2614102320 2024-04-28T12:49:21.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2612ffd700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102320 0x7f261419f7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f260400c920 tx=0x7f260400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:21.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604007b00 con 0x7f2614102320 2024-04-28T12:49:21.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261419a3b0 con 0x7f2614102320 2024-04-28T12:49:21.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2604004ce0 con 0x7f2614102320 2024-04-28T12:49:21.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604005500 con 0x7f2614102320 2024-04-28T12:49:21.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.284+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26141a6670 con 0x7f2614102320 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.285+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2614106410 con 0x7f2614102320 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.286+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2604007ca0 con 0x7f2614102320 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f25fbfff700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f25fc0a3cd0 0x7f25fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f26127fc700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f25fc0a3cd0 0x7f25fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f26040d0090 con 0x7f2614102320 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f25fbfff700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] conn(0x7f25fc0ad4b0 0x7f25fc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 --> [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f260400eec0 con 0x7f25fc0ad4b0 2024-04-28T12:49:21.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f26040d7050 con 0x7f2614102320 2024-04-28T12:49:21.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f26127fc700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f25fc0a3cd0 0x7f25fc0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2608006fd0 tx=0x7f2608008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:21.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.287+0000 7f26137fe700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] conn(0x7f25fc0ad4b0 0x7f25fc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:21.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.288+0000 7f26137fe700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] conn(0x7f25fc0ad4b0 0x7f25fc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:21.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.289+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== osd.7 v2:172.21.15.77:6816/1250050488 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f260400eec0 con 0x7f25fc0ad4b0 2024-04-28T12:49:21.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.306+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 --> [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2614103a40 con 0x7f25fc0ad4b0 2024-04-28T12:49:21.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.306+0000 7f25fbfff700 1 -- 172.21.15.53:0/1057905525 <== osd.7 v2:172.21.15.77:6816/1250050488 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2614103a40 con 0x7f25fc0ad4b0 2024-04-28T12:49:21.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] conn(0x7f25fc0ad4b0 msgr2=0x7f25fc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:21.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] conn(0x7f25fc0ad4b0 0x7f25fc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:21.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f25fc0a3cd0 msgr2=0x7f25fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:21.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f25fc0a3cd0 0x7f25fc0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2608006fd0 tx=0x7f2608008040 comp rx=0 tx=0).stop 2024-04-28T12:49:21.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102320 msgr2=0x7f261419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:21.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 --2- 172.21.15.53:0/1057905525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2614102320 0x7f261419f7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f260400c920 tx=0x7f260400cc30 comp rx=0 tx=0).stop 2024-04-28T12:49:21.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f26137fe700 1 -- 172.21.15.53:0/1057905525 reap_dead start 2024-04-28T12:49:21.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 shutdown_connections 2024-04-28T12:49:21.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 >> 172.21.15.53:0/1057905525 conn(0x7f26140792d0 msgr2=0x7f26141015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:21.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 shutdown_connections 2024-04-28T12:49:21.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:21.307+0000 7f2619624700 1 -- 172.21.15.53:0/1057905525 wait complete. 2024-04-28T12:49:21.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:21 smithi053 ceph-mon[29182]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T12:49:21.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:21 smithi077 ceph-mon[32553]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T12:49:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 -- 172.21.15.53:0/1571416550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c1076b0 msgr2=0x7f8c4c10f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 --2- 172.21.15.53:0/1571416550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c1076b0 0x7f8c4c10f250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400b0d0 tx=0x7f8c4400b3e0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 -- 172.21.15.53:0/1571416550 shutdown_connections 2024-04-28T12:49:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 --2- 172.21.15.53:0/1571416550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c1076b0 0x7f8c4c10f250 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 --2- 172.21.15.53:0/1571416550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c4c102960 0x7f8c4c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 --2- 172.21.15.53:0/1571416550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c4c103310 0x7f8c4c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 -- 172.21.15.53:0/1571416550 >> 172.21.15.53:0/1571416550 conn(0x7f8c4c0792d0 msgr2=0x7f8c4c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:22.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 -- 172.21.15.53:0/1571416550 shutdown_connections 2024-04-28T12:49:22.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.029+0000 7f8c50db3700 1 -- 172.21.15.53:0/1571416550 wait complete. 2024-04-28T12:49:22.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 Processor -- start 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 -- start start 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c102960 0x7f8c4c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c4c103310 0x7f8c4c19af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c4c1076b0 0x7f8c4c1a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1125a0 con 0x7f8c4c102960 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c112420 con 0x7f8c4c1076b0 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c50db3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c112720 con 0x7f8c4c103310 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c102960 0x7f8c4c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c102960 0x7f8c4c19aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48198/0 (socket says 172.21.15.53:48198) 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 -- 172.21.15.53:0/68130230 learned_addr learned my addr 172.21.15.53:0/68130230 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4ad9d700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c4c1076b0 0x7f8c4c1a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c49d9b700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c4c103310 0x7f8c4c19af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 -- 172.21.15.53:0/68130230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c4c103310 msgr2=0x7f8c4c19af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c4c103310 0x7f8c4c19af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 -- 172.21.15.53:0/68130230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c4c1076b0 msgr2=0x7f8c4c1a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c4c1076b0 0x7f8c4c1a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.030+0000 7f8c4a59c700 1 -- 172.21.15.53:0/68130230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c1a5fc0 con 0x7f8c4c102960 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.031+0000 7f8c4a59c700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c102960 0x7f8c4c19aa00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00b530 tx=0x7f8c3c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.031+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c015020 con 0x7f8c4c102960 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.031+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c1a6190 con 0x7f8c4c102960 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.031+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c3c011ce0 con 0x7f8c4c102960 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.031+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c010d20 con 0x7f8c4c102960 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.031+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c1a6720 con 0x7f8c4c102960 2024-04-28T12:49:22.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.032+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8c4c063b30 con 0x7f8c4c102960 2024-04-28T12:49:22.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.033+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8c3c018020 con 0x7f8c4c102960 2024-04-28T12:49:22.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c3b7fe700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8c340a3cd0 0x7f8c340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c49d9b700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8c340a3cd0 0x7f8c340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8c3c0d0360 con 0x7f8c4c102960 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c3b7fe700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] conn(0x7f8c340ad4b0 0x7f8c340af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 --> [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c3c010ec0 con 0x7f8c340ad4b0 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f8c3c0d0710 con 0x7f8c4c102960 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c4ad9d700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] conn(0x7f8c340ad4b0 0x7f8c340af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c49d9b700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8c340a3cd0 0x7f8c340a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8c40005950 tx=0x7f8c400058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.034+0000 7f8c4ad9d700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] conn(0x7f8c340ad4b0 0x7f8c340af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:22.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.035+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== osd.3 v2:172.21.15.53:6810/3076698859 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c3c010ec0 con 0x7f8c340ad4b0 2024-04-28T12:49:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:21 smithi191 ceph-mon[33016]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T12:49:22.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 --> [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8c4c063f30 con 0x7f8c340ad4b0 2024-04-28T12:49:22.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c3b7fe700 1 -- 172.21.15.53:0/68130230 <== osd.3 v2:172.21.15.53:6810/3076698859 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8c4c063f30 con 0x7f8c340ad4b0 2024-04-28T12:49:22.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] conn(0x7f8c340ad4b0 msgr2=0x7f8c340af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] conn(0x7f8c340ad4b0 0x7f8c340af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8c340a3cd0 msgr2=0x7f8c340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8c340a3cd0 0x7f8c340a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8c40005950 tx=0x7f8c400058e0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c102960 msgr2=0x7f8c4c19aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 --2- 172.21.15.53:0/68130230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c4c102960 0x7f8c4c19aa00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00b530 tx=0x7f8c3c00b8f0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c4ad9d700 1 -- 172.21.15.53:0/68130230 reap_dead start 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.052+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 shutdown_connections 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.053+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 >> 172.21.15.53:0/68130230 conn(0x7f8c4c0792d0 msgr2=0x7f8c4c108800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.053+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 shutdown_connections 2024-04-28T12:49:22.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.053+0000 7f8c50db3700 1 -- 172.21.15.53:0/68130230 wait complete. 2024-04-28T12:49:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.567+0000 7f3fe9837700 1 -- 172.21.15.53:0/717369534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103470 msgr2=0x7f3fe41038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.567+0000 7f3fe9837700 1 --2- 172.21.15.53:0/717369534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103470 0x7f3fe41038f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009990 tx=0x7f3fcc009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 -- 172.21.15.53:0/717369534 shutdown_connections 2024-04-28T12:49:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 --2- 172.21.15.53:0/717369534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fe4103e30 0x7f3fe4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 --2- 172.21.15.53:0/717369534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fe41094c0 0x7f3fe41098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 --2- 172.21.15.53:0/717369534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103470 0x7f3fe41038f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 -- 172.21.15.53:0/717369534 >> 172.21.15.53:0/717369534 conn(0x7f3fe407a580 msgr2=0x7f3fe407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 -- 172.21.15.53:0/717369534 shutdown_connections 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 -- 172.21.15.53:0/717369534 wait complete. 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 Processor -- start 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.568+0000 7f3fe9837700 1 -- start start 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe9837700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fe4103470 0x7f3fe4075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe9837700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103e30 0x7f3fe4075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe9837700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fe41094c0 0x7f3fe40761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe9837700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe4113640 con 0x7f3fe4103e30 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe9837700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41134c0 con 0x7f3fe41094c0 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe9837700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41137c0 con 0x7f3fe4103470 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103e30 0x7f3fe4075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103e30 0x7f3fe4075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48220/0 (socket says 172.21.15.53:48220) 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 -- 172.21.15.53:0/2250781174 learned_addr learned my addr 172.21.15.53:0/2250781174 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe2ffd700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fe4103470 0x7f3fe4075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fe37fe700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fe41094c0 0x7f3fe40761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 -- 172.21.15.53:0/2250781174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fe4103470 msgr2=0x7f3fe4075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fe4103470 0x7f3fe4075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 -- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fe41094c0 msgr2=0x7f3fe40761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fe41094c0 0x7f3fe40761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 -- 172.21.15.53:0/2250781174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe41a8120 con 0x7f3fe4103e30 2024-04-28T12:49:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.569+0000 7f3fdbfff700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103e30 0x7f3fe4075c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc000c00 tx=0x7f3fcc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.570+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc027070 con 0x7f3fe4103e30 2024-04-28T12:49:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.570+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe41a83b0 con 0x7f3fe4103e30 2024-04-28T12:49:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.570+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fcc0040e0 con 0x7f3fe4103e30 2024-04-28T12:49:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.570+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc00f460 con 0x7f3fe4103e30 2024-04-28T12:49:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.570+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe41a88e0 con 0x7f3fe4103e30 2024-04-28T12:49:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.571+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3fe4063b30 con 0x7f3fe4103e30 2024-04-28T12:49:22.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.572+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3fcc02a070 con 0x7f3fe4103e30 2024-04-28T12:49:22.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.573+0000 7f3fe8835700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3fd00a3cd0 0x7f3fd00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.573+0000 7f3fe2ffd700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3fd00a3cd0 0x7f3fd00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.573+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3fcc0d1a80 con 0x7f3fe4103e30 2024-04-28T12:49:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.573+0000 7f3fe8835700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] conn(0x7f3fd00ad4b0 0x7f3fd00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.573+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 --> [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3fcc011ec0 con 0x7f3fd00ad4b0 2024-04-28T12:49:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.573+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f3fcc0d1f60 con 0x7f3fe4103e30 2024-04-28T12:49:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.573+0000 7f3fe2ffd700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3fd00a3cd0 0x7f3fd00a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009a10 tx=0x7f3fd4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.574+0000 7f3fe37fe700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] conn(0x7f3fd00ad4b0 0x7f3fd00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.574+0000 7f3fe37fe700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] conn(0x7f3fd00ad4b0 0x7f3fd00af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.575+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== osd.4 v2:172.21.15.77:6808/3997878075 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3fcc011ec0 con 0x7f3fd00ad4b0 2024-04-28T12:49:22.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.591+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 --> [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3fe4063f30 con 0x7f3fd00ad4b0 2024-04-28T12:49:22.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe8835700 1 -- 172.21.15.53:0/2250781174 <== osd.4 v2:172.21.15.77:6808/3997878075 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3fe4063f30 con 0x7f3fd00ad4b0 2024-04-28T12:49:22.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] conn(0x7f3fd00ad4b0 msgr2=0x7f3fd00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] conn(0x7f3fd00ad4b0 0x7f3fd00af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3fd00a3cd0 msgr2=0x7f3fd00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3fd00a3cd0 0x7f3fd00a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009a10 tx=0x7f3fd4009190 comp rx=0 tx=0).stop 2024-04-28T12:49:22.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103e30 msgr2=0x7f3fe4075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:22.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 --2- 172.21.15.53:0/2250781174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fe4103e30 0x7f3fe4075c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc000c00 tx=0x7f3fcc0047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:22.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe37fe700 1 -- 172.21.15.53:0/2250781174 reap_dead start 2024-04-28T12:49:22.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 shutdown_connections 2024-04-28T12:49:22.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 >> 172.21.15.53:0/2250781174 conn(0x7f3fe407a580 msgr2=0x7f3fe4102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:22.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 shutdown_connections 2024-04-28T12:49:22.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:22.592+0000 7f3fe9837700 1 -- 172.21.15.53:0/2250781174 wait complete. 2024-04-28T12:49:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1510169069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806baf0 msgr2=0x7f5a7806bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 --2- 172.21.15.53:0/1510169069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806baf0 0x7f5a7806bf70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009990 tx=0x7f5a6c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1510169069 shutdown_connections 2024-04-28T12:49:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 --2- 172.21.15.53:0/1510169069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a7806c4b0 0x7f5a781102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 --2- 172.21.15.53:0/1510169069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a781094a0 0x7f5a78109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 --2- 172.21.15.53:0/1510169069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806baf0 0x7f5a7806bf70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1510169069 >> 172.21.15.53:0/1510169069 conn(0x7f5a780792d0 msgr2=0x7f5a780796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1510169069 shutdown_connections 2024-04-28T12:49:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1510169069 wait complete. 2024-04-28T12:49:23.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.095+0000 7f5a7d8aa700 1 Processor -- start 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a7d8aa700 1 -- start start 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a7d8aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a7806baf0 0x7f5a781027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a7d8aa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806c4b0 0x7f5a78102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a7d8aa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a781094a0 0x7f5a78103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a7d8aa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5a781135d0 con 0x7f5a7806c4b0 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a7d8aa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a78113450 con 0x7f5a781094a0 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a7d8aa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a78113750 con 0x7f5a7806baf0 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806c4b0 0x7f5a78102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:23.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806c4b0 0x7f5a78102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48246/0 (socket says 172.21.15.53:48246) 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 -- 172.21.15.53:0/1372370518 learned_addr learned my addr 172.21.15.53:0/1372370518 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a76ffd700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a7806baf0 0x7f5a781027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a777fe700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a781094a0 0x7f5a78103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 -- 172.21.15.53:0/1372370518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a7806baf0 msgr2=0x7f5a781027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a7806baf0 0x7f5a781027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 -- 172.21.15.53:0/1372370518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a781094a0 msgr2=0x7f5a78103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a781094a0 0x7f5a78103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.096+0000 7f5a767fc700 1 -- 172.21.15.53:0/1372370518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a781a8030 con 0x7f5a7806c4b0 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.097+0000 7f5a767fc700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806c4b0 0x7f5a78102d10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c000c00 tx=0x7f5a6c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.097+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c027070 con 0x7f5a7806c4b0 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.097+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6c003f20 con 0x7f5a7806c4b0 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.097+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a781a82c0 con 0x7f5a7806c4b0 2024-04-28T12:49:23.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.097+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c022ca0 con 0x7f5a7806c4b0 2024-04-28T12:49:23.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.097+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a781a8710 con 0x7f5a7806c4b0 2024-04-28T12:49:23.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.098+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a78063b30 con 0x7f5a7806c4b0 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.099+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a6c02a070 con 0x7f5a7806c4b0 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a7c8a8700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a600a3cd0 0x7f5a600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a76ffd700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a600a3cd0 0x7f5a600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5a6c0d1c70 con 0x7f5a7806c4b0 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a7c8a8700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] conn(0x7f5a600ad4b0 0x7f5a600af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 --> [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a6c0095d0 con 0x7f5a600ad4b0 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f5a6c0d2150 con 0x7f5a7806c4b0 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a777fe700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] conn(0x7f5a600ad4b0 0x7f5a600af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:23.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a76ffd700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a600a3cd0 0x7f5a600a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5a680045b0 tx=0x7f5a6800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:23.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.100+0000 7f5a777fe700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] conn(0x7f5a600ad4b0 0x7f5a600af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:23.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.101+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== osd.0 v2:172.21.15.53:6802/1076361705 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a6c0095d0 con 0x7f5a600ad4b0 2024-04-28T12:49:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.121+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 --> [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5a78063fb0 con 0x7f5a600ad4b0 2024-04-28T12:49:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a7c8a8700 1 -- 172.21.15.53:0/1372370518 <== osd.0 v2:172.21.15.53:6802/1076361705 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5a78063fb0 con 0x7f5a600ad4b0 2024-04-28T12:49:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] conn(0x7f5a600ad4b0 msgr2=0x7f5a600af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a7d8aa700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] conn(0x7f5a600ad4b0 0x7f5a600af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a600a3cd0 msgr2=0x7f5a600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a7d8aa700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a600a3cd0 0x7f5a600a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5a680045b0 tx=0x7f5a6800b3c0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806c4b0 msgr2=0x7f5a78102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a7d8aa700 1 --2- 172.21.15.53:0/1372370518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a7806c4b0 0x7f5a78102d10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c000c00 tx=0x7f5a6c0047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:23.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.122+0000 7f5a777fe700 1 -- 172.21.15.53:0/1372370518 reap_dead start 2024-04-28T12:49:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.123+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 shutdown_connections 2024-04-28T12:49:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.123+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 >> 172.21.15.53:0/1372370518 conn(0x7f5a780792d0 msgr2=0x7f5a78101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.123+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 shutdown_connections 2024-04-28T12:49:23.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:23.123+0000 7f5a7d8aa700 1 -- 172.21.15.53:0/1372370518 wait complete. 2024-04-28T12:49:23.869 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:23 smithi053 ceph-mon[29182]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:23.899 INFO:teuthology.orchestra.run.smithi053.stdout:111669149759 2024-04-28T12:49:23.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-28T12:49:23.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:23 smithi077 ceph-mon[32553]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:24.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:23 smithi191 ceph-mon[33016]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:24.438 INFO:teuthology.orchestra.run.smithi053.stdout:34359738446 2024-04-28T12:49:24.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-28T12:49:25.631 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:25 smithi053 ceph-mon[29182]: pgmap v311: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:25.663 INFO:teuthology.orchestra.run.smithi053.stdout:201863462958 2024-04-28T12:49:25.663 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-28T12:49:25.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:25 smithi077 ceph-mon[32553]: pgmap v311: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:25 smithi191 ceph-mon[33016]: pgmap v311: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:26.281 INFO:teuthology.orchestra.run.smithi053.stdout:223338299434 2024-04-28T12:49:26.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-28T12:49:27.757 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:27 smithi053 ceph-mon[29182]: pgmap v312: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:27.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 -- 172.21.15.53:0/2969172073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1034a0 msgr2=0x7f58cc103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 --2- 172.21.15.53:0/2969172073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1034a0 0x7f58cc103920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009990 tx=0x7f58b8009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 -- 172.21.15.53:0/2969172073 shutdown_connections 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 --2- 172.21.15.53:0/2969172073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc103e60 0x7f58cc1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 --2- 172.21.15.53:0/2969172073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58cc1094e0 0x7f58cc1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 --2- 172.21.15.53:0/2969172073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1034a0 0x7f58cc103920 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 -- 172.21.15.53:0/2969172073 >> 172.21.15.53:0/2969172073 conn(0x7f58cc07a590 msgr2=0x7f58cc07a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 -- 172.21.15.53:0/2969172073 shutdown_connections 2024-04-28T12:49:27.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.866+0000 7f58d302a700 1 -- 172.21.15.53:0/2969172073 wait complete. 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 Processor -- start 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 -- start start 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc1034a0 0x7f58cc19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58cc103e60 0x7f58cc19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1094e0 0x7f58cc199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc113660 con 0x7f58cc1094e0 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1134e0 con 0x7f58cc103e60 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d302a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc1137e0 con 0x7f58cc1034a0 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d15c7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1094e0 0x7f58cc199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d15c7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1094e0 0x7f58cc199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34098/0 (socket says 172.21.15.53:34098) 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d0dc6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc1034a0 0x7f58cc19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58d15c7700 1 -- 172.21.15.53:0/1298892078 learned_addr learned my addr 172.21.15.53:0/1298892078 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.867+0000 7f58cbfff700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58cc103e60 0x7f58cc19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d15c7700 1 -- 172.21.15.53:0/1298892078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc1034a0 msgr2=0x7f58cc19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d15c7700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58cc1034a0 0x7f58cc19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d15c7700 1 -- 172.21.15.53:0/1298892078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58cc103e60 msgr2=0x7f58cc19fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d15c7700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58cc103e60 0x7f58cc19fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d15c7700 1 -- 172.21.15.53:0/1298892078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58cc19a0c0 con 0x7f58cc1094e0 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58cbfff700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58cc103e60 0x7f58cc19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d15c7700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1094e0 0x7f58cc199960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f58c400b970 tx=0x7f58c400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4019070 con 0x7f58cc1094e0 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58cc19a3b0 con 0x7f58cc1094e0 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58c400cde0 con 0x7f58cc1094e0 2024-04-28T12:49:27.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4007370 con 0x7f58cc1094e0 2024-04-28T12:49:27.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.868+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58cc1a6670 con 0x7f58cc1094e0 2024-04-28T12:49:27.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.869+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f58cc063b70 con 0x7f58cc1094e0 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.870+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f58c401c070 con 0x7f58cc1094e0 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58c9ffb700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f58bc0a3cd0 0x7f58bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58d0dc6700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f58bc0a3cd0 0x7f58bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f58c40d01f0 con 0x7f58cc1094e0 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58c9ffb700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] conn(0x7f58bc0ad4b0 0x7f58bc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 --> [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58c4020e70 con 0x7f58bc0ad4b0 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f58c40d06d0 con 0x7f58cc1094e0 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58d0dc6700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f58bc0a3cd0 0x7f58bc0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f58cc19aaa0 tx=0x7f58c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:27.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.871+0000 7f58cbfff700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] conn(0x7f58bc0ad4b0 0x7f58bc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:27.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.872+0000 7f58cbfff700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] conn(0x7f58bc0ad4b0 0x7f58bc0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:27.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.873+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== osd.8 v2:172.21.15.191:6816/1864030319 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58c4020e70 con 0x7f58bc0ad4b0 2024-04-28T12:49:27.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.889+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 --> [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f58cc063ff0 con 0x7f58bc0ad4b0 2024-04-28T12:49:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58c9ffb700 1 -- 172.21.15.53:0/1298892078 <== osd.8 v2:172.21.15.191:6816/1864030319 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f58cc063ff0 con 0x7f58bc0ad4b0 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 >> [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] conn(0x7f58bc0ad4b0 msgr2=0x7f58bc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58d302a700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] conn(0x7f58bc0ad4b0 0x7f58bc0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f58bc0a3cd0 msgr2=0x7f58bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58d302a700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f58bc0a3cd0 0x7f58bc0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f58cc19aaa0 tx=0x7f58c000b410 comp rx=0 tx=0).stop 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1094e0 msgr2=0x7f58cc199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58d302a700 1 --2- 172.21.15.53:0/1298892078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc1094e0 0x7f58cc199960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f58c400b970 tx=0x7f58c400bc80 comp rx=0 tx=0).stop 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.890+0000 7f58d15c7700 1 -- 172.21.15.53:0/1298892078 reap_dead start 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.891+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 shutdown_connections 2024-04-28T12:49:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.891+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 >> 172.21.15.53:0/1298892078 conn(0x7f58cc07a590 msgr2=0x7f58cc102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:27.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.891+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 shutdown_connections 2024-04-28T12:49:27.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:27.891+0000 7f58d302a700 1 -- 172.21.15.53:0/1298892078 wait complete. 2024-04-28T12:49:27.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:27 smithi077 ceph-mon[32553]: pgmap v312: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:27 smithi191 ceph-mon[33016]: pgmap v312: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:28.456 INFO:teuthology.orchestra.run.smithi053.stdout:146028888122 2024-04-28T12:49:28.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-28T12:49:28.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.693+0000 7f9cd8983700 1 -- 172.21.15.53:0/15647807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4102310 msgr2=0x7f9cd4102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.693+0000 7f9cd8983700 1 --2- 172.21.15.53:0/15647807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4102310 0x7f9cd4102790 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8009990 tx=0x7f9cc8009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.693+0000 7f9cd8983700 1 -- 172.21.15.53:0/15647807 shutdown_connections 2024-04-28T12:49:28.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.693+0000 7f9cd8983700 1 --2- 172.21.15.53:0/15647807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cd4102cd0 0x7f9cd410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.694+0000 7f9cd8983700 1 --2- 172.21.15.53:0/15647807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd4108380 0x7f9cd4108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.694+0000 7f9cd8983700 1 --2- 172.21.15.53:0/15647807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4102310 0x7f9cd4102790 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.694+0000 7f9cd8983700 1 -- 172.21.15.53:0/15647807 >> 172.21.15.53:0/15647807 conn(0x7f9cd4079290 msgr2=0x7f9cd40796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:28.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.694+0000 7f9cd8983700 1 -- 172.21.15.53:0/15647807 shutdown_connections 2024-04-28T12:49:28.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.694+0000 7f9cd8983700 1 -- 172.21.15.53:0/15647807 wait complete. 2024-04-28T12:49:28.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.695+0000 7f9cd8983700 1 Processor -- start 2024-04-28T12:49:28.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.695+0000 7f9cd8983700 1 -- start start 2024-04-28T12:49:28.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd8983700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd4102310 0x7f9cd4109240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd8983700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cd4102cd0 0x7f9cd4109780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd8983700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4108380 0x7f9cd4109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd8983700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd41124c0 con 0x7f9cd4108380 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd8983700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd4112340 con 0x7f9cd4102cd0 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd8983700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd4112640 con 0x7f9cd4102310 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cd4102cd0 0x7f9cd4109780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4108380 0x7f9cd4109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4108380 0x7f9cd4109cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34116/0 (socket says 172.21.15.53:34116) 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cd4102cd0 0x7f9cd4109780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.53:48224/0 (socket says 172.21.15.53:48224) 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd1d9b700 1 -- 172.21.15.53:0/1441833818 learned_addr learned my addr 172.21.15.53:0/1441833818 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd259c700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd4102310 0x7f9cd4109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd1d9b700 1 -- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd4102310 msgr2=0x7f9cd4109240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd1d9b700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd4102310 0x7f9cd4109240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd1d9b700 1 -- 172.21.15.53:0/1441833818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4108380 msgr2=0x7f9cd4109cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.696+0000 7f9cd1d9b700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cd4108380 0x7f9cd4109cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.697+0000 7f9cd1d9b700 1 -- 172.21.15.53:0/1441833818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd41ac410 con 0x7f9cd4102cd0 2024-04-28T12:49:28.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.697+0000 7f9cd259c700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd4102310 0x7f9cd4109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:28.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.697+0000 7f9cd1d9b700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cd4102cd0 0x7f9cd4109780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8005f50 tx=0x7f9cc80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:28.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.697+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8027070 con 0x7f9cd4102cd0 2024-04-28T12:49:28.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.697+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cc8003f20 con 0x7f9cd4102cd0 2024-04-28T12:49:28.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.697+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd41ac6a0 con 0x7f9cd4102cd0 2024-04-28T12:49:28.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.697+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd41acb50 con 0x7f9cd4102cd0 2024-04-28T12:49:28.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.698+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8022c30 con 0x7f9cd4102cd0 2024-04-28T12:49:28.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.698+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9cb4001030 con 0x7f9cd4102cd0 2024-04-28T12:49:28.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.700+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9cc802a070 con 0x7f9cd4102cd0 2024-04-28T12:49:28.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.701+0000 7f9cc37fe700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9cbc0a3dd0 0x7f9cbc0a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.701+0000 7f9cd259c700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9cbc0a3dd0 0x7f9cbc0a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.701+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9cc80d11f0 con 0x7f9cd4102cd0 2024-04-28T12:49:28.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.701+0000 7f9cc37fe700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] conn(0x7f9cbc0ad5b0 0x7f9cbc0af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.701+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 --> [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cc8009030 con 0x7f9cbc0ad5b0 2024-04-28T12:49:28.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.701+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f9cc80d15e0 con 0x7f9cd4102cd0 2024-04-28T12:49:28.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.701+0000 7f9cd259c700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9cbc0a3dd0 0x7f9cbc0a6290 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4005950 tx=0x7f9cc400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:28.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.702+0000 7f9cd2d9d700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] conn(0x7f9cbc0ad5b0 0x7f9cbc0af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.702+0000 7f9cd2d9d700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] conn(0x7f9cbc0ad5b0 0x7f9cbc0af9d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:28.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.703+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== osd.5 v2:172.21.15.191:6808/2749306755 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc8009030 con 0x7f9cbc0ad5b0 2024-04-28T12:49:28.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.720+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 --> [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9cb4002e60 con 0x7f9cbc0ad5b0 2024-04-28T12:49:28.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.720+0000 7f9cc37fe700 1 -- 172.21.15.53:0/1441833818 <== osd.5 v2:172.21.15.191:6808/2749306755 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9cb4002e60 con 0x7f9cbc0ad5b0 2024-04-28T12:49:28.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] conn(0x7f9cbc0ad5b0 msgr2=0x7f9cbc0af9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] conn(0x7f9cbc0ad5b0 0x7f9cbc0af9d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9cbc0a3dd0 msgr2=0x7f9cbc0a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9cbc0a3dd0 0x7f9cbc0a6290 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4005950 tx=0x7f9cc400a300 comp rx=0 tx=0).stop 2024-04-28T12:49:28.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cd4102cd0 msgr2=0x7f9cd4109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 --2- 172.21.15.53:0/1441833818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cd4102cd0 0x7f9cd4109780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8005f50 tx=0x7f9cc80047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd2d9d700 1 -- 172.21.15.53:0/1441833818 reap_dead start 2024-04-28T12:49:28.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 shutdown_connections 2024-04-28T12:49:28.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 >> 172.21.15.53:0/1441833818 conn(0x7f9cd4079290 msgr2=0x7f9cd4101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:28.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 shutdown_connections 2024-04-28T12:49:28.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:28.721+0000 7f9cd8983700 1 -- 172.21.15.53:0/1441833818 wait complete. 2024-04-28T12:49:29.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.242+0000 7f3bf86fd700 1 -- 172.21.15.53:0/1905998703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102ce0 msgr2=0x7f3bf010f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.242+0000 7f3bf86fd700 1 --2- 172.21.15.53:0/1905998703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102ce0 0x7f3bf010f1d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3be800b0a0 tx=0x7f3be800b3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- 172.21.15.53:0/1905998703 shutdown_connections 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 --2- 172.21.15.53:0/1905998703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102ce0 0x7f3bf010f1d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 --2- 172.21.15.53:0/1905998703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf0108390 0x7f3bf0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 --2- 172.21.15.53:0/1905998703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bf0102320 0x7f3bf01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- 172.21.15.53:0/1905998703 >> 172.21.15.53:0/1905998703 conn(0x7f3bf00792d0 msgr2=0x7f3bf00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- 172.21.15.53:0/1905998703 shutdown_connections 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- 172.21.15.53:0/1905998703 wait complete. 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 Processor -- start 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- start start 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102320 0x7f3bf019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf0102ce0 0x7f3bf019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bf0108390 0x7f3bf01998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf0113630 con 0x7f3bf0102320 2024-04-28T12:49:29.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf01134b0 con 0x7f3bf0108390 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.243+0000 7f3bf86fd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf01137b0 con 0x7f3bf0102ce0 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102320 0x7f3bf019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102320 0x7f3bf019f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34136/0 (socket says 172.21.15.53:34136) 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 -- 172.21.15.53:0/3280447107 learned_addr learned my addr 172.21.15.53:0/3280447107 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6c9a700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bf0108390 0x7f3bf01998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf5c98700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf0102ce0 0x7f3bf019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 -- 172.21.15.53:0/3280447107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf0102ce0 msgr2=0x7f3bf019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf0102ce0 0x7f3bf019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 -- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bf0108390 msgr2=0x7f3bf01998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bf0108390 0x7f3bf01998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 -- 172.21.15.53:0/3280447107 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf019a030 con 0x7f3bf0102320 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6c9a700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bf0108390 0x7f3bf01998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:29.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf6499700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102320 0x7f3bf019f710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3be000e9c0 tx=0x7f3be000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:29.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be000cbd0 con 0x7f3bf0102320 2024-04-28T12:49:29.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3be000cd70 con 0x7f3bf0102320 2024-04-28T12:49:29.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf019a320 con 0x7f3bf0102320 2024-04-28T12:49:29.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0010430 con 0x7f3bf0102320 2024-04-28T12:49:29.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.244+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf01a65e0 con 0x7f3bf0102320 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.245+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3bf0063b70 con 0x7f3bf0102320 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.247+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3be0004d70 con 0x7f3bf0102320 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.247+0000 7f3be77fe700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3bdc0a3cd0 0x7f3bdc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.247+0000 7f3bf5c98700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3bdc0a3cd0 0x7f3bdc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.247+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3be00d0b50 con 0x7f3bf0102320 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.247+0000 7f3be77fe700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] conn(0x7f3bdc0ad4b0 0x7f3bdc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.248+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 --> [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3be0017e50 con 0x7f3bdc0ad4b0 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.248+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f3be00d1030 con 0x7f3bf0102320 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.248+0000 7f3bf5c98700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3bdc0a3cd0 0x7f3bdc0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3bec005fd0 tx=0x7f3bec005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:29.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.248+0000 7f3bf6c9a700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] conn(0x7f3bdc0ad4b0 0x7f3bdc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.248+0000 7f3bf6c9a700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] conn(0x7f3bdc0ad4b0 0x7f3bdc0af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:29.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.249+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== osd.10 v2:172.21.15.77:6824/2556585572 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3be0017e50 con 0x7f3bdc0ad4b0 2024-04-28T12:49:29.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.266+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 --> [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3bf0063ff0 con 0x7f3bdc0ad4b0 2024-04-28T12:49:29.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.266+0000 7f3be77fe700 1 -- 172.21.15.53:0/3280447107 <== osd.10 v2:172.21.15.77:6824/2556585572 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3bf0063ff0 con 0x7f3bdc0ad4b0 2024-04-28T12:49:29.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] conn(0x7f3bdc0ad4b0 msgr2=0x7f3bdc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] conn(0x7f3bdc0ad4b0 0x7f3bdc0af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3bdc0a3cd0 msgr2=0x7f3bdc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3bdc0a3cd0 0x7f3bdc0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3bec005fd0 tx=0x7f3bec005f60 comp rx=0 tx=0).stop 2024-04-28T12:49:29.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102320 msgr2=0x7f3bf019f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 --2- 172.21.15.53:0/3280447107 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bf0102320 0x7f3bf019f710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3be000e9c0 tx=0x7f3be000ed80 comp rx=0 tx=0).stop 2024-04-28T12:49:29.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf6c9a700 1 -- 172.21.15.53:0/3280447107 reap_dead start 2024-04-28T12:49:29.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 shutdown_connections 2024-04-28T12:49:29.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 >> 172.21.15.53:0/3280447107 conn(0x7f3bf00792d0 msgr2=0x7f3bf0101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:29.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 shutdown_connections 2024-04-28T12:49:29.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.267+0000 7f3bf86fd700 1 -- 172.21.15.53:0/3280447107 wait complete. 2024-04-28T12:49:29.890 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:29 smithi053 ceph-mon[29182]: pgmap v313: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:29.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 -- 172.21.15.53:0/2237464208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31381075b0 msgr2=0x7f313810f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 --2- 172.21.15.53:0/2237464208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31381075b0 0x7f313810f180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f313000b0d0 tx=0x7f313000b3e0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 -- 172.21.15.53:0/2237464208 shutdown_connections 2024-04-28T12:49:29.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 --2- 172.21.15.53:0/2237464208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31381075b0 0x7f313810f180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 --2- 172.21.15.53:0/2237464208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31381024d0 0x7f31381028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 --2- 172.21.15.53:0/2237464208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3138102e80 0x7f3138106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 -- 172.21.15.53:0/2237464208 >> 172.21.15.53:0/2237464208 conn(0x7f31380fdda0 msgr2=0x7f31381001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:29.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 -- 172.21.15.53:0/2237464208 shutdown_connections 2024-04-28T12:49:29.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.901+0000 7f313d5ff700 1 -- 172.21.15.53:0/2237464208 wait complete. 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 Processor -- start 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 -- start start 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31381024d0 0x7f313819f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3138102e80 0x7f313819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31381075b0 0x7f3138199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3138112470 con 0x7f3138102e80 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f31381122f0 con 0x7f31381075b0 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f313d5ff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f31381125f0 con 0x7f31381024d0 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3138102e80 0x7f313819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3138102e80 0x7f313819fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34160/0 (socket says 172.21.15.53:34160) 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 -- 172.21.15.53:0/3531693253 learned_addr learned my addr 172.21.15.53:0/3531693253 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f3136ffd700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31381024d0 0x7f313819f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f31377fe700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31381075b0 0x7f3138199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 -- 172.21.15.53:0/3531693253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31381024d0 msgr2=0x7f313819f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31381024d0 0x7f313819f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 -- 172.21.15.53:0/3531693253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31381075b0 msgr2=0x7f3138199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31381075b0 0x7f3138199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.902+0000 7f312ffff700 1 -- 172.21.15.53:0/3531693253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313819a020 con 0x7f3138102e80 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.903+0000 7f312ffff700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3138102e80 0x7f313819fbb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f312000d950 tx=0x7f312000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.903+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120017070 con 0x7f3138102e80 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.903+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3120005020 con 0x7f3138102e80 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.903+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f313819a310 con 0x7f3138102e80 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.903+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312000fa10 con 0x7f3138102e80 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.903+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31381a64d0 con 0x7f3138102e80 2024-04-28T12:49:29.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.904+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3138063bb0 con 0x7f3138102e80 2024-04-28T12:49:29.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.905+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3120004a30 con 0x7f3138102e80 2024-04-28T12:49:29.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f3134ff9700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f31240a3cd0 0x7f31240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f3136ffd700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f31240a3cd0 0x7f31240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f31200d00c0 con 0x7f3138102e80 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f3134ff9700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] conn(0x7f31240ad4b0 0x7f31240af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 --> [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f312001ce50 con 0x7f31240ad4b0 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f31200d0470 con 0x7f3138102e80 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f31377fe700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] conn(0x7f31240ad4b0 0x7f31240af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.906+0000 7f3136ffd700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f31240a3cd0 0x7f31240a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3128009c60 tx=0x7f3128009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.907+0000 7f31377fe700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] conn(0x7f31240ad4b0 0x7f31240af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:29.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.907+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== osd.9 v2:172.21.15.53:6826/3727947776 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f312001ce50 con 0x7f31240ad4b0 2024-04-28T12:49:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.923+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 --> [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3138064030 con 0x7f31240ad4b0 2024-04-28T12:49:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f3134ff9700 1 -- 172.21.15.53:0/3531693253 <== osd.9 v2:172.21.15.53:6826/3727947776 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3138064030 con 0x7f31240ad4b0 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] conn(0x7f31240ad4b0 msgr2=0x7f31240af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] conn(0x7f31240ad4b0 0x7f31240af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f31240a3cd0 msgr2=0x7f31240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f31240a3cd0 0x7f31240a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3128009c60 tx=0x7f3128009260 comp rx=0 tx=0).stop 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3138102e80 msgr2=0x7f313819fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 --2- 172.21.15.53:0/3531693253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3138102e80 0x7f313819fbb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f312000d950 tx=0x7f312000dc60 comp rx=0 tx=0).stop 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f31377fe700 1 -- 172.21.15.53:0/3531693253 reap_dead start 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 shutdown_connections 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 >> 172.21.15.53:0/3531693253 conn(0x7f31380fdda0 msgr2=0x7f313806b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 shutdown_connections 2024-04-28T12:49:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:29.924+0000 7f313d5ff700 1 -- 172.21.15.53:0/3531693253 wait complete. 2024-04-28T12:49:29.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:29 smithi077 ceph-mon[32553]: pgmap v313: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:30.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:29 smithi191 ceph-mon[33016]: pgmap v313: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:30.263 INFO:teuthology.orchestra.run.smithi053.stdout:257698037799 2024-04-28T12:49:30.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-28T12:49:30.749 INFO:teuthology.orchestra.run.smithi053.stdout:313532612634 2024-04-28T12:49:30.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-28T12:49:31.298 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:31.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:31 smithi053 ceph-mon[29182]: pgmap v314: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:31.671 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:31.675 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:31.728 INFO:teuthology.orchestra.run.smithi053.stdout:292057776162 2024-04-28T12:49:31.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-28T12:49:31.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:31 smithi077 ceph-mon[32553]: pgmap v314: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:31 smithi191 ceph-mon[33016]: pgmap v314: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:32.122 INFO:teuthology.orchestra.run.smithi053.stdout:167503724597 2024-04-28T12:49:32.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-28T12:49:32.383 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:32.383 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:33.465 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:33.466 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:33.466 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:33.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:33 smithi053 ceph-mon[29182]: pgmap v315: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T12:49:33.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:33 smithi077 ceph-mon[32553]: pgmap v315: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T12:49:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:33 smithi191 ceph-mon[33016]: pgmap v315: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T12:49:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-28T12:49:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-28T12:49:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-28T12:49:34.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-28T12:49:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-28T12:49:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-28T12:49:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-28T12:49:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-28T12:49:34.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:34 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-28T12:49:34.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-28T12:49:34.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:34 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-28T12:49:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:34 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-28T12:49:35.370 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: pgmap v316: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-28T12:49:35.791 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-28T12:49:35.792 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:35 smithi053 ceph-mon[29182]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T12:49:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: pgmap v316: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-28T12:49:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-28T12:49:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-28T12:49:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-28T12:49:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-28T12:49:35.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-28T12:49:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-28T12:49:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-28T12:49:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-28T12:49:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:35 smithi077 ceph-mon[32553]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T12:49:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: pgmap v316: 97 pgs: 97 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-28T12:49:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-28T12:49:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-28T12:49:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-28T12:49:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-28T12:49:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-28T12:49:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-28T12:49:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-28T12:49:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-28T12:49:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:35 smithi191 ceph-mon[33016]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T12:49:36.674 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:36 smithi053 ceph-mon[29182]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T12:49:36.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:36 smithi077 ceph-mon[32553]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T12:49:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:36 smithi191 ceph-mon[33016]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T12:49:37.481 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:37.481 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:37.481 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:37.738 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:37 smithi053 ceph-mon[29182]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:37.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:37 smithi077 ceph-mon[32553]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:37 smithi191 ceph-mon[33016]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:39.733 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:39 smithi053 ceph-mon[29182]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:39.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:39 smithi077 ceph-mon[32553]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:39 smithi191 ceph-mon[33016]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:41.753 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:41 smithi053 ceph-mon[29182]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:41.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:41 smithi077 ceph-mon[32553]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:41 smithi191 ceph-mon[33016]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:42.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 -- 172.21.15.53:0/747932822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 msgr2=0x7fd368109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/747932822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd368109870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd35c009990 tx=0x7fd35c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 -- 172.21.15.53:0/747932822 shutdown_connections 2024-04-28T12:49:42.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/747932822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd368109870 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/747932822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806c360 0x7fd3681102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/747932822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd36806b9c0 0x7fd36806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 -- 172.21.15.53:0/747932822 >> 172.21.15.53:0/747932822 conn(0x7fd368079290 msgr2=0x7fd3680796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:42.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 -- 172.21.15.53:0/747932822 shutdown_connections 2024-04-28T12:49:42.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.052+0000 7fd36cbcd700 1 -- 172.21.15.53:0/747932822 wait complete. 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 Processor -- start 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 -- start start 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806b9c0 0x7fd36819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd36806c360 0x7fd36819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd3681a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd3681135a0 con 0x7fd368109490 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd368113420 con 0x7fd36806c360 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36cbcd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd368113720 con 0x7fd36806b9c0 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd3681a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806b9c0 0x7fd36819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd36806c360 0x7fd36819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806b9c0 0x7fd36819bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.53:49810/0 (socket says 172.21.15.53:49810) 2024-04-28T12:49:42.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd36659c700 1 -- 172.21.15.53:0/792584524 learned_addr learned my addr 172.21.15.53:0/792584524 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd3681a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54422/0 (socket says 172.21.15.53:54422) 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd366d9d700 1 -- 172.21.15.53:0/792584524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806b9c0 msgr2=0x7fd36819bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd366d9d700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806b9c0 0x7fd36819bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd366d9d700 1 -- 172.21.15.53:0/792584524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd36806c360 msgr2=0x7fd36819c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.053+0000 7fd366d9d700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd36806c360 0x7fd36819c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd366d9d700 1 -- 172.21.15.53:0/792584524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3681a0d20 con 0x7fd368109490 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd36659c700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806b9c0 0x7fd36819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd365d9b700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd36806c360 0x7fd36819c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd366d9d700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd3681a05e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd36000ab40 tx=0x7fd36000ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd3577fe700 1 -- 172.21.15.53:0/792584524 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360017020 con 0x7fd368109490 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd3577fe700 1 -- 172.21.15.53:0/792584524 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd360004bb0 con 0x7fd368109490 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3681a1010 con 0x7fd368109490 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd3577fe700 1 -- 172.21.15.53:0/792584524 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360005280 con 0x7fd368109490 2024-04-28T12:49:42.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.054+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3681a8890 con 0x7fd368109490 2024-04-28T12:49:42.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.055+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36806a110 con 0x7fd368109490 2024-04-28T12:49:42.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.056+0000 7fd3577fe700 1 -- 172.21.15.53:0/792584524 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd36000c580 con 0x7fd368109490 2024-04-28T12:49:42.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.057+0000 7fd3577fe700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd3500a3d50 0x7fd3500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.057+0000 7fd36659c700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd3500a3d50 0x7fd3500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.057+0000 7fd3577fe700 1 -- 172.21.15.53:0/792584524 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fd3600cffc0 con 0x7fd368109490 2024-04-28T12:49:42.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.058+0000 7fd36659c700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd3500a3d50 0x7fd3500a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd35c003800 tx=0x7fd35c005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:42.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.060+0000 7fd3577fe700 1 -- 172.21.15.53:0/792584524 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd360021020 con 0x7fd368109490 2024-04-28T12:49:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.283+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd36819d070 con 0x7fd368109490 2024-04-28T12:49:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.284+0000 7fd3577fe700 1 -- 172.21.15.53:0/792584524 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd36009c8d0 con 0x7fd368109490 2024-04-28T12:49:42.280 INFO:teuthology.orchestra.run.smithi053.stdout:167503724600 2024-04-28T12:49:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd3500a3d50 msgr2=0x7fd3500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd3500a3d50 0x7fd3500a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd35c003800 tx=0x7fd35c005c00 comp rx=0 tx=0).stop 2024-04-28T12:49:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 msgr2=0x7fd3681a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd3681a05e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd36000ab40 tx=0x7fd36000ae50 comp rx=0 tx=0).stop 2024-04-28T12:49:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 shutdown_connections 2024-04-28T12:49:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd3500a3d50 0x7fd3500a6210 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd368109490 0x7fd3681a05e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd36806b9c0 0x7fd36819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 --2- 172.21.15.53:0/792584524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd36806c360 0x7fd36819c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 >> 172.21.15.53:0/792584524 conn(0x7fd368079290 msgr2=0x7fd36810e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 shutdown_connections 2024-04-28T12:49:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.286+0000 7fd36cbcd700 1 -- 172.21.15.53:0/792584524 wait complete. 2024-04-28T12:49:42.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 -- 172.21.15.53:0/1415364789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874102960 msgr2=0x7f6874102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1415364789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874102960 0x7f6874102d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f68680099c0 tx=0x7f6868009cd0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 -- 172.21.15.53:0/1415364789 shutdown_connections 2024-04-28T12:49:42.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1415364789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874102960 0x7f6874102d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1415364789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68741076b0 0x7f687410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1415364789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6874103310 0x7f6874106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 -- 172.21.15.53:0/1415364789 >> 172.21.15.53:0/1415364789 conn(0x7f68740792d0 msgr2=0x7f68740796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 -- 172.21.15.53:0/1415364789 shutdown_connections 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.959+0000 7f687ad8d700 1 -- 172.21.15.53:0/1415364789 wait complete. 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 Processor -- start 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 -- start start 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6874102960 0x7f687419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874103310 0x7f687419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68741076b0 0x7f68741998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6874112510 con 0x7f6874103310 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6874112390 con 0x7f6874102960 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687ad8d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6874112690 con 0x7f68741076b0 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f6873fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874103310 0x7f687419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f6873fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874103310 0x7f687419fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54404/0 (socket says 172.21.15.53:54404) 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f6873fff700 1 -- 172.21.15.53:0/1894549351 learned_addr learned my addr 172.21.15.53:0/1894549351 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f6878b29700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6874102960 0x7f687419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.960+0000 7f687932a700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68741076b0 0x7f68741998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.681 INFO:teuthology.orchestra.run.smithi053.stdout:201863462962 2024-04-28T12:49:42.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6873fff700 1 -- 172.21.15.53:0/1894549351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68741076b0 msgr2=0x7f68741998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6873fff700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68741076b0 0x7f68741998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6873fff700 1 -- 172.21.15.53:0/1894549351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6874102960 msgr2=0x7f687419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6873fff700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6874102960 0x7f687419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6873fff700 1 -- 172.21.15.53:0/1894549351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687419a020 con 0x7f6874103310 2024-04-28T12:49:42.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6873fff700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874103310 0x7f687419fc40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f686000e940 tx=0x7f686000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6871ffb700 1 -- 172.21.15.53:0/1894549351 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686000cbd0 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6871ffb700 1 -- 172.21.15.53:0/1894549351 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f686000ee90 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f6871ffb700 1 -- 172.21.15.53:0/1894549351 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6860018680 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687419a310 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.961+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68741a65d0 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.962+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6874103a60 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.964+0000 7f6871ffb700 1 -- 172.21.15.53:0/1894549351 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6860010430 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.964+0000 7f6871ffb700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68640a3cd0 0x7f68640a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.964+0000 7f6871ffb700 1 -- 172.21.15.53:0/1894549351 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f68600d0b10 con 0x7f6874103310 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.964+0000 7f6878b29700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68640a3cd0 0x7f68640a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.965+0000 7f6878b29700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68640a3cd0 0x7f68640a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6868000c00 tx=0x7f6868005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:41.968+0000 7f6871ffb700 1 -- 172.21.15.53:0/1894549351 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f686009d390 con 0x7f6874103310 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.182+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f687419ae20 con 0x7f6874103310 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.182+0000 7f6871ffb700 1 -- 172.21.15.53:0/1894549351 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f687419ae20 con 0x7f6874103310 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68640a3cd0 msgr2=0x7f68640a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68640a3cd0 0x7f68640a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6868000c00 tx=0x7f6868005fb0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874103310 msgr2=0x7f687419fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874103310 0x7f687419fc40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f686000e940 tx=0x7f686000ec50 comp rx=0 tx=0).stop 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 shutdown_connections 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68640a3cd0 0x7f68640a6190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68741076b0 0x7f68741998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6874102960 0x7f687419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 --2- 172.21.15.53:0/1894549351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6874103310 0x7f687419fc40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 >> 172.21.15.53:0/1894549351 conn(0x7f68740792d0 msgr2=0x7f687406bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 shutdown_connections 2024-04-28T12:49:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.185+0000 7f687ad8d700 1 -- 172.21.15.53:0/1894549351 wait complete. 2024-04-28T12:49:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:42 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1894549351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T12:49:42.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:42 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/792584524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T12:49:42.947 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:42 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1894549351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T12:49:42.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:42 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/792584524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T12:49:43.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:42 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1894549351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T12:49:43.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:42 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/792584524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T12:49:43.445 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462958 got 201863462962 for osd.6 2024-04-28T12:49:43.446 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 -- 172.21.15.53:0/4131156602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 msgr2=0x7f85a0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 --2- 172.21.15.53:0/4131156602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0109880 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8594009990 tx=0x7f8594009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 -- 172.21.15.53:0/4131156602 shutdown_connections 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 --2- 172.21.15.53:0/4131156602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0109880 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 --2- 172.21.15.53:0/4131156602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a006bad0 0x7f85a01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 --2- 172.21.15.53:0/4131156602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85a006b130 0x7f85a006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 -- 172.21.15.53:0/4131156602 >> 172.21.15.53:0/4131156602 conn(0x7f85a00792d0 msgr2=0x7f85a00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 -- 172.21.15.53:0/4131156602 shutdown_connections 2024-04-28T12:49:43.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 -- 172.21.15.53:0/4131156602 wait complete. 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 Processor -- start 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.689+0000 7f85a71d3700 1 -- start start 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a71d3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85a006b130 0x7f85a01023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a71d3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a006bad0 0x7f85a0102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a71d3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a71d3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85a0113600 con 0x7f85a01094a0 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a71d3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85a0113480 con 0x7f85a006bad0 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a71d3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f85a0113780 con 0x7f85a006b130 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54444/0 (socket says 172.21.15.53:54444) 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 -- 172.21.15.53:0/54831031 learned_addr learned my addr 172.21.15.53:0/54831031 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f859ffff700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a006bad0 0x7f85a0102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:43.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a4f6f700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85a006b130 0x7f85a01023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:43.698 INFO:teuthology.orchestra.run.smithi053.stdout:34359738450 2024-04-28T12:49:43.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 -- 172.21.15.53:0/54831031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85a006b130 msgr2=0x7f85a01023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:43.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85a006b130 0x7f85a01023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 -- 172.21.15.53:0/54831031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a006bad0 msgr2=0x7f85a0102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:43.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a006bad0 0x7f85a0102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 -- 172.21.15.53:0/54831031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a01a8060 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a4f6f700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85a006b130 0x7f85a01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f85a5770700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0102e70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8598007d70 tx=0x7f859800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.690+0000 7f859dffb700 1 -- 172.21.15.53:0/54831031 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598016020 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.691+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a01a8350 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.691+0000 7f859dffb700 1 -- 172.21.15.53:0/54831031 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f859800ad40 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.691+0000 7f859dffb700 1 -- 172.21.15.53:0/54831031 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85980115e0 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.691+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a01a88e0 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.692+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a006a110 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.693+0000 7f859dffb700 1 -- 172.21.15.53:0/54831031 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8598019020 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.693+0000 7f859dffb700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85900a3cd0 0x7f85900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.694+0000 7f85a4f6f700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85900a3cd0 0x7f85900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.694+0000 7f859dffb700 1 -- 172.21.15.53:0/54831031 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f85980d0380 con 0x7f85a01094a0 2024-04-28T12:49:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.694+0000 7f85a4f6f700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85900a3cd0 0x7f85900a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f859400b5c0 tx=0x7f8594005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.698+0000 7f859dffb700 1 -- 172.21.15.53:0/54831031 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8598020020 con 0x7f85a01094a0 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.910+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f85a002e490 con 0x7f85a01094a0 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.910+0000 7f859dffb700 1 -- 172.21.15.53:0/54831031 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f859809cd10 con 0x7f85a01094a0 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.912+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85900a3cd0 msgr2=0x7f85900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.912+0000 7f85a71d3700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85900a3cd0 0x7f85900a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f859400b5c0 tx=0x7f8594005fb0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.912+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 msgr2=0x7f85a0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.912+0000 7f85a71d3700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0102e70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8598007d70 tx=0x7f859800a340 comp rx=0 tx=0).stop 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.912+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 shutdown_connections 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.913+0000 7f85a71d3700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85900a3cd0 0x7f85900a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.913+0000 7f85a71d3700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a01094a0 0x7f85a0102e70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.913+0000 7f85a71d3700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85a006b130 0x7f85a01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.913+0000 7f85a71d3700 1 --2- 172.21.15.53:0/54831031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a006bad0 0x7f85a0102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.913+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 >> 172.21.15.53:0/54831031 conn(0x7f85a00792d0 msgr2=0x7f85a01004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.913+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 shutdown_connections 2024-04-28T12:49:43.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:42.913+0000 7f85a71d3700 1 -- 172.21.15.53:0/54831031 wait complete. 2024-04-28T12:49:43.701 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:43 smithi053 ceph-mon[29182]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:43.701 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:43 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:49:43.701 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/54831031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T12:49:43.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:43 smithi077 ceph-mon[32553]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:43.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:43 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:49:43.828 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:43 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/54831031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T12:49:43.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:43 smithi191 ceph-mon[33016]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 6.2 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:43.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:43 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:49:43.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:43 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/54831031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T12:49:45.780 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.780 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:45 smithi053 ceph-mon[29182]: pgmap v323: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 5.0 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:45.780 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.780 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.780 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:45 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:45 smithi191 ceph-mon[33016]: pgmap v323: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 5.0 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:45 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:45 smithi077 ceph-mon[32553]: pgmap v323: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 5.0 KiB/s, 1 keys/s, 1 objects/s recovering 2024-04-28T12:49:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:45.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:45 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:49:47.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 -- 172.21.15.53:0/2299383832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 msgr2=0x7f0430108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 --2- 172.21.15.53:0/2299383832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430108760 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0420009990 tx=0x7f0420009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 -- 172.21.15.53:0/2299383832 shutdown_connections 2024-04-28T12:49:47.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 --2- 172.21.15.53:0/2299383832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430108760 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 --2- 172.21.15.53:0/2299383832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0430102cd0 0x7f043010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 --2- 172.21.15.53:0/2299383832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0430102310 0x7f0430102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 -- 172.21.15.53:0/2299383832 >> 172.21.15.53:0/2299383832 conn(0x7f0430079290 msgr2=0x7f04300796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 -- 172.21.15.53:0/2299383832 shutdown_connections 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.331+0000 7f0434d5f700 1 -- 172.21.15.53:0/2299383832 wait complete. 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 Processor -- start 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 -- start start 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0430102310 0x7f043019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0430102cd0 0x7f043019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0430112520 con 0x7f0430108380 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f04301123a0 con 0x7f0430102cd0 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0434d5f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f04301126a0 con 0x7f0430102310 2024-04-28T12:49:47.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34356/0 (socket says 172.21.15.53:34356) 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 -- 172.21.15.53:0/4212491440 learned_addr learned my addr 172.21.15.53:0/4212491440 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f0427fff700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0430102cd0 0x7f043019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042e59c700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0430102310 0x7f043019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 -- 172.21.15.53:0/4212491440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0430102310 msgr2=0x7f043019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0430102310 0x7f043019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 -- 172.21.15.53:0/4212491440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0430102cd0 msgr2=0x7f043019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0430102cd0 0x7f043019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042ed9d700 1 -- 172.21.15.53:0/4212491440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043019a0c0 con 0x7f0430108380 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.332+0000 7f042e59c700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0430102310 0x7f043019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.333+0000 7f042ed9d700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430199960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0428007ee0 tx=0x7f042800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.333+0000 7f04277fe700 1 -- 172.21.15.53:0/4212491440 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042800dbd0 con 0x7f0430108380 2024-04-28T12:49:47.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.333+0000 7f04277fe700 1 -- 172.21.15.53:0/4212491440 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f042800dd70 con 0x7f0430108380 2024-04-28T12:49:47.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.333+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043019a3b0 con 0x7f0430108380 2024-04-28T12:49:47.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.333+0000 7f04277fe700 1 -- 172.21.15.53:0/4212491440 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428005990 con 0x7f0430108380 2024-04-28T12:49:47.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.333+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04301a6620 con 0x7f0430108380 2024-04-28T12:49:47.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.334+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043006a110 con 0x7f0430108380 2024-04-28T12:49:47.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.336+0000 7f04277fe700 1 -- 172.21.15.53:0/4212491440 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f042801a020 con 0x7f0430108380 2024-04-28T12:49:47.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.337+0000 7f04277fe700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f041c0a3d50 0x7f041c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.337+0000 7f042e59c700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f041c0a3d50 0x7f041c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.337+0000 7f04277fe700 1 -- 172.21.15.53:0/4212491440 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f04280d0350 con 0x7f0430108380 2024-04-28T12:49:47.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.338+0000 7f042e59c700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f041c0a3d50 0x7f041c0a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f042000b5c0 tx=0x7f0420005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:47.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.340+0000 7f04277fe700 1 -- 172.21.15.53:0/4212491440 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0428010070 con 0x7f0430108380 2024-04-28T12:49:47.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.560+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f04301a68b0 con 0x7f0430108380 2024-04-28T12:49:47.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.560+0000 7f04277fe700 1 -- 172.21.15.53:0/4212491440 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f042809cbd0 con 0x7f0430108380 2024-04-28T12:49:47.557 INFO:teuthology.orchestra.run.smithi053.stdout:347892351002 2024-04-28T12:49:47.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f041c0a3d50 msgr2=0x7f041c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f041c0a3d50 0x7f041c0a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f042000b5c0 tx=0x7f0420005fb0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 msgr2=0x7f0430199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430199960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0428007ee0 tx=0x7f042800d3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 shutdown_connections 2024-04-28T12:49:47.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f041c0a3d50 0x7f041c0a6210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0430108380 0x7f0430199960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0430102310 0x7f043019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 --2- 172.21.15.53:0/4212491440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0430102cd0 0x7f043019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 >> 172.21.15.53:0/4212491440 conn(0x7f0430079290 msgr2=0x7f043006bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:47.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 shutdown_connections 2024-04-28T12:49:47.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.563+0000 7f0434d5f700 1 -- 172.21.15.53:0/4212491440 wait complete. 2024-04-28T12:49:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:47 smithi191 ceph-mon[33016]: pgmap v324: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 4.5 KiB/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:47 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1298274816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T12:49:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:47 smithi053 ceph-mon[29182]: pgmap v324: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 4.5 KiB/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:47.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1298274816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T12:49:47.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:47 smithi077 ceph-mon[32553]: pgmap v324: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 4.5 KiB/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:47.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:47 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1298274816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T12:49:48.239 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350995 got 347892351002 for osd.11 2024-04-28T12:49:48.240 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:48.382 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738450 for osd.0 2024-04-28T12:49:48.385 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:48 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4212491440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T12:49:48.882 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:48 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4212491440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T12:49:48.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:48 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4212491440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T12:49:49.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 -- 172.21.15.53:0/3118086051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301094f0 msgr2=0x7f5a301098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 --2- 172.21.15.53:0/3118086051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301094f0 0x7f5a301098d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009990 tx=0x7f5a18009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 -- 172.21.15.53:0/3118086051 shutdown_connections 2024-04-28T12:49:49.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 --2- 172.21.15.53:0/3118086051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301094f0 0x7f5a301098d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 --2- 172.21.15.53:0/3118086051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a30103e90 0x7f5a301102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 --2- 172.21.15.53:0/3118086051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a301034d0 0x7f5a30103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 -- 172.21.15.53:0/3118086051 >> 172.21.15.53:0/3118086051 conn(0x7f5a3007a500 msgr2=0x7f5a3007a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 -- 172.21.15.53:0/3118086051 shutdown_connections 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.172+0000 7f5a35442700 1 -- 172.21.15.53:0/3118086051 wait complete. 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 Processor -- start 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 -- start start 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301034d0 0x7f5a3019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a30103e90 0x7f5a3019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a301094f0 0x7f5a30199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5a30113620 con 0x7f5a301034d0 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301134a0 con 0x7f5a301094f0 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a35442700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301137a0 con 0x7f5a30103e90 2024-04-28T12:49:49.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.173+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301034d0 0x7f5a3019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301034d0 0x7f5a3019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34404/0 (socket says 172.21.15.53:34404) 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 -- 172.21.15.53:0/1902746252 learned_addr learned my addr 172.21.15.53:0/1902746252 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2f7fe700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a301094f0 0x7f5a30199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2e7fc700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a30103e90 0x7f5a3019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 -- 172.21.15.53:0/1902746252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a30103e90 msgr2=0x7f5a3019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a30103e90 0x7f5a3019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 -- 172.21.15.53:0/1902746252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a301094f0 msgr2=0x7f5a30199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a301094f0 0x7f5a30199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 -- 172.21.15.53:0/1902746252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3019a0c0 con 0x7f5a301034d0 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a2effd700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301034d0 0x7f5a3019f750 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a18009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a27fff700 1 -- 172.21.15.53:0/1902746252 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18027070 con 0x7f5a301034d0 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3019a350 con 0x7f5a301034d0 2024-04-28T12:49:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a27fff700 1 -- 172.21.15.53:0/1902746252 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a18022470 con 0x7f5a301034d0 2024-04-28T12:49:49.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a27fff700 1 -- 172.21.15.53:0/1902746252 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18017650 con 0x7f5a301034d0 2024-04-28T12:49:49.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.174+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a301a6620 con 0x7f5a301034d0 2024-04-28T12:49:49.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.175+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3010b960 con 0x7f5a301034d0 2024-04-28T12:49:49.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.178+0000 7f5a27fff700 1 -- 172.21.15.53:0/1902746252 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a1802a070 con 0x7f5a301034d0 2024-04-28T12:49:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.179+0000 7f5a27fff700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.179+0000 7f5a2e7fc700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.179+0000 7f5a27fff700 1 -- 172.21.15.53:0/1902746252 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f5a180d14c0 con 0x7f5a301034d0 2024-04-28T12:49:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.179+0000 7f5a2e7fc700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5a30108530 tx=0x7f5a200073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:49.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.182+0000 7f5a27fff700 1 -- 172.21.15.53:0/1902746252 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a18013070 con 0x7f5a301034d0 2024-04-28T12:49:49.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.394+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5a3019ae40 con 0x7f5a301034d0 2024-04-28T12:49:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.394+0000 7f5a27fff700 1 -- 172.21.15.53:0/1902746252 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5a1809ddd0 con 0x7f5a301034d0 2024-04-28T12:49:49.391 INFO:teuthology.orchestra.run.smithi053.stdout:77309411400 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a1c0a3cd0 msgr2=0x7f5a1c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5a30108530 tx=0x7f5a200073d0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301034d0 msgr2=0x7f5a3019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301034d0 0x7f5a3019f750 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a18009580 comp rx=0 tx=0).stop 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 shutdown_connections 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5a1c0a3cd0 0x7f5a1c0a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a301034d0 0x7f5a3019f750 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a301094f0 0x7f5a30199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 --2- 172.21.15.53:0/1902746252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a30103e90 0x7f5a3019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 >> 172.21.15.53:0/1902746252 conn(0x7f5a3007a500 msgr2=0x7f5a30102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:49.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 shutdown_connections 2024-04-28T12:49:49.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.397+0000 7f5a35442700 1 -- 172.21.15.53:0/1902746252 wait complete. 2024-04-28T12:49:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:49 smithi191 ceph-mon[33016]: pgmap v325: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 4.1 KiB/s, 0 objects/s recovering 2024-04-28T12:49:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:49 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2753341720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T12:49:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:49 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1902746252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T12:49:49.811 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:49 smithi053 ceph-mon[29182]: pgmap v325: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 4.1 KiB/s, 0 objects/s recovering 2024-04-28T12:49:49.811 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:49 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2753341720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T12:49:49.811 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:49 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1902746252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T12:49:49.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:49 smithi077 ceph-mon[32553]: pgmap v325: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 4.1 KiB/s, 0 objects/s recovering 2024-04-28T12:49:49.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:49 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2753341720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T12:49:49.849 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:49 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1902746252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T12:49:49.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2864947557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d81094f0 msgr2=0x7f85d81098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2864947557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d81094f0 0x7f85d81098d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009990 tx=0x7f85c8009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2864947557 shutdown_connections 2024-04-28T12:49:49.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2864947557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d81094f0 0x7f85d81098d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2864947557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d8103e90 0x7f85d81102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2864947557 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d81034d0 0x7f85d8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2864947557 >> 172.21.15.53:0/2864947557 conn(0x7f85d807a4b0 msgr2=0x7f85d807a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2864947557 shutdown_connections 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2864947557 wait complete. 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.079+0000 7f85ddc0c700 1 Processor -- start 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85ddc0c700 1 -- start start 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85ddc0c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d81034d0 0x7f85d819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85ddc0c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d8103e90 0x7f85d819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85ddc0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d81094f0 0x7f85d81998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85ddc0c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85d81135f0 con 0x7f85d8103e90 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85ddc0c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85d8113470 con 0x7f85d81094f0 2024-04-28T12:49:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85ddc0c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f85d8113770 con 0x7f85d81034d0 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d8103e90 0x7f85d819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d8103e90 0x7f85d819fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34378/0 (socket says 172.21.15.53:34378) 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 -- 172.21.15.53:0/2753341720 learned_addr learned my addr 172.21.15.53:0/2753341720 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d77fe700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d81034d0 0x7f85d819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d7fff700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d81094f0 0x7f85d81998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 -- 172.21.15.53:0/2753341720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d81034d0 msgr2=0x7f85d819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d81034d0 0x7f85d819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 -- 172.21.15.53:0/2753341720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d81094f0 msgr2=0x7f85d81998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d81094f0 0x7f85d81998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d6ffd700 1 -- 172.21.15.53:0/2753341720 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d819a030 con 0x7f85d8103e90 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.080+0000 7f85d77fe700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d81034d0 0x7f85d819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.081+0000 7f85d6ffd700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d8103e90 0x7f85d819fc50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00d980 tx=0x7f85cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.081+0000 7f85d4ff9700 1 -- 172.21.15.53:0/2753341720 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc016070 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.081+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d819a320 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.081+0000 7f85d4ff9700 1 -- 172.21.15.53:0/2753341720 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85cc004830 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.081+0000 7f85d4ff9700 1 -- 172.21.15.53:0/2753341720 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc0054c0 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.081+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d81a65e0 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.082+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d806a110 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.083+0000 7f85d4ff9700 1 -- 172.21.15.53:0/2753341720 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f85cc004220 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.084+0000 7f85d4ff9700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85c00a3cd0 0x7f85c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.084+0000 7f85d4ff9700 1 -- 172.21.15.53:0/2753341720 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f85cc0d0390 con 0x7f85d8103e90 2024-04-28T12:49:49.934 INFO:teuthology.orchestra.run.smithi053.stdout:111669149764 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.084+0000 7f85d77fe700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85c00a3cd0 0x7f85c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.084+0000 7f85d77fe700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85c00a3cd0 0x7f85c00a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f85c800b5c0 tx=0x7f85c8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.088+0000 7f85d4ff9700 1 -- 172.21.15.53:0/2753341720 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f85cc01f020 con 0x7f85d8103e90 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.304+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f85d802e490 con 0x7f85d8103e90 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.305+0000 7f85d4ff9700 1 -- 172.21.15.53:0/2753341720 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f85cc09cc10 con 0x7f85d8103e90 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85c00a3cd0 msgr2=0x7f85c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85c00a3cd0 0x7f85c00a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f85c800b5c0 tx=0x7f85c8005960 comp rx=0 tx=0).stop 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d8103e90 msgr2=0x7f85d819fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d8103e90 0x7f85d819fc50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00d980 tx=0x7f85cc00dc90 comp rx=0 tx=0).stop 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 shutdown_connections 2024-04-28T12:49:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f85c00a3cd0 0x7f85c00a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d81094f0 0x7f85d81998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85d81034d0 0x7f85d819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 --2- 172.21.15.53:0/2753341720 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85d8103e90 0x7f85d819fc50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 >> 172.21.15.53:0/2753341720 conn(0x7f85d807a4b0 msgr2=0x7f85d8102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 shutdown_connections 2024-04-28T12:49:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:49.307+0000 7f85ddc0c700 1 -- 172.21.15.53:0/2753341720 wait complete. 2024-04-28T12:49:50.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 -- 172.21.15.53:0/334297671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c1094c0 msgr2=0x7f428c1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 --2- 172.21.15.53:0/334297671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c1094c0 0x7f428c1098a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4288009990 tx=0x7f4288009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 -- 172.21.15.53:0/334297671 shutdown_connections 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 --2- 172.21.15.53:0/334297671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c1094c0 0x7f428c1098a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 --2- 172.21.15.53:0/334297671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c103e30 0x7f428c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 --2- 172.21.15.53:0/334297671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c103470 0x7f428c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 -- 172.21.15.53:0/334297671 >> 172.21.15.53:0/334297671 conn(0x7f428c07a580 msgr2=0x7f428c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 -- 172.21.15.53:0/334297671 shutdown_connections 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.226+0000 7f429434f700 1 -- 172.21.15.53:0/334297671 wait complete. 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 Processor -- start 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 -- start start 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c103470 0x7f428c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c103e30 0x7f428c19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c1094c0 0x7f428c199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f428c113640 con 0x7f428c103470 2024-04-28T12:49:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f428c1134c0 con 0x7f428c1094c0 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f429434f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f428c1137c0 con 0x7f428c103e30 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f42920eb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c103470 0x7f428c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f42920eb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c103470 0x7f428c19f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34338/0 (socket says 172.21.15.53:34338) 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.227+0000 7f42920eb700 1 -- 172.21.15.53:0/1298274816 learned_addr learned my addr 172.21.15.53:0/1298274816 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42928ec700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c1094c0 0x7f428c199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42918ea700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c103e30 0x7f428c19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42920eb700 1 -- 172.21.15.53:0/1298274816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c103e30 msgr2=0x7f428c19fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42920eb700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c103e30 0x7f428c19fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42920eb700 1 -- 172.21.15.53:0/1298274816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c1094c0 msgr2=0x7f428c199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42920eb700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c1094c0 0x7f428c199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42920eb700 1 -- 172.21.15.53:0/1298274816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c19a0e0 con 0x7f428c103470 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42918ea700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c103e30 0x7f428c19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42920eb700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c103470 0x7f428c19f7c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f42880057c0 tx=0x7f4288009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42837fe700 1 -- 172.21.15.53:0/1298274816 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288026070 con 0x7f428c103470 2024-04-28T12:49:50.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428c19a370 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42837fe700 1 -- 172.21.15.53:0/1298274816 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f428800fbf0 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f42837fe700 1 -- 172.21.15.53:0/1298274816 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288021e60 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.228+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428c075650 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.229+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428c104b60 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.231+0000 7f42837fe700 1 -- 172.21.15.53:0/1298274816 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4288029070 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.231+0000 7f42837fe700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42780a3cd0 0x7f42780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.231+0000 7f42837fe700 1 -- 172.21.15.53:0/1298274816 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f42880d1360 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.231+0000 7f42918ea700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42780a3cd0 0x7f42780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.232+0000 7f42918ea700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42780a3cd0 0x7f42780a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f428c07ac60 tx=0x7f427c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.236+0000 7f42837fe700 1 -- 172.21.15.53:0/1298274816 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4288013070 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.456+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f428c02e490 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.456+0000 7f42837fe700 1 -- 172.21.15.53:0/1298274816 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f428809dc70 con 0x7f428c103470 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42780a3cd0 msgr2=0x7f42780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42780a3cd0 0x7f42780a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f428c07ac60 tx=0x7f427c009260 comp rx=0 tx=0).stop 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c103470 msgr2=0x7f428c19f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c103470 0x7f428c19f7c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f42880057c0 tx=0x7f4288009580 comp rx=0 tx=0).stop 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 shutdown_connections 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42780a3cd0 0x7f42780a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f428c103470 0x7f428c19f7c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f428c1094c0 0x7f428c199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 --2- 172.21.15.53:0/1298274816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f428c103e30 0x7f428c19fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 >> 172.21.15.53:0/1298274816 conn(0x7f428c07a580 msgr2=0x7f428c102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 shutdown_connections 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:47.459+0000 7f429434f700 1 -- 172.21.15.53:0/1298274816 wait complete. 2024-04-28T12:49:50.246 INFO:teuthology.orchestra.run.smithi053.stdout:223338299439 2024-04-28T12:49:51.190 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724597 got 167503724600 for osd.5 2024-04-28T12:49:51.190 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:51 smithi191 ceph-mon[33016]: pgmap v326: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 4.1 KiB/s, 0 objects/s recovering 2024-04-28T12:49:51.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:51 smithi053 ceph-mon[29182]: pgmap v326: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 4.1 KiB/s, 0 objects/s recovering 2024-04-28T12:49:51.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:51 smithi077 ceph-mon[32553]: pgmap v326: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 4.1 KiB/s, 0 objects/s recovering 2024-04-28T12:49:52.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 -- 172.21.15.53:0/3895390392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103470 msgr2=0x7f4c081038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/3895390392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103470 0x7f4c081038f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009990 tx=0x7f4bfc009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 -- 172.21.15.53:0/3895390392 shutdown_connections 2024-04-28T12:49:52.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/3895390392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c08103e30 0x7f4c08110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/3895390392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c081094c0 0x7f4c081098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/3895390392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103470 0x7f4c081038f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 -- 172.21.15.53:0/3895390392 >> 172.21.15.53:0/3895390392 conn(0x7f4c0807a580 msgr2=0x7f4c0807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.080+0000 7f4c0f91a700 1 -- 172.21.15.53:0/3895390392 shutdown_connections 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 -- 172.21.15.53:0/3895390392 wait complete. 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 Processor -- start 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 -- start start 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c08103470 0x7f4c0819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103e30 0x7f4c0819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c081094c0 0x7f4c08199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081135e0 con 0x7f4c08103e30 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c08113460 con 0x7f4c08103470 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.081+0000 7f4c0f91a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4c08113760 con 0x7f4c081094c0 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0ceb5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103e30 0x7f4c0819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0d6b6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c08103470 0x7f4c0819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0d6b6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c08103470 0x7f4c0819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.53:60564/0 (socket says 172.21.15.53:60564) 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0d6b6700 1 -- 172.21.15.53:0/2052523685 learned_addr learned my addr 172.21.15.53:0/2052523685 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0deb7700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c081094c0 0x7f4c08199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0ceb5700 1 -- 172.21.15.53:0/2052523685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c081094c0 msgr2=0x7f4c08199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0ceb5700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c081094c0 0x7f4c08199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0ceb5700 1 -- 172.21.15.53:0/2052523685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c08103470 msgr2=0x7f4c0819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0ceb5700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c08103470 0x7f4c0819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0ceb5700 1 -- 172.21.15.53:0/2052523685 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0819a0a0 con 0x7f4c08103e30 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0deb7700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c081094c0 0x7f4c08199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0ceb5700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103e30 0x7f4c0819fcc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0059d0 tx=0x7f4bfc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c067fc700 1 -- 172.21.15.53:0/2052523685 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc027070 con 0x7f4c08103e30 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c067fc700 1 -- 172.21.15.53:0/2052523685 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bfc003fe0 con 0x7f4c08103e30 2024-04-28T12:49:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0819a330 con 0x7f4c08103e30 2024-04-28T12:49:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c067fc700 1 -- 172.21.15.53:0/2052523685 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc022bb0 con 0x7f4c08103e30 2024-04-28T12:49:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.082+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c081a6650 con 0x7f4c08103e30 2024-04-28T12:49:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.083+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c08104b60 con 0x7f4c08103e30 2024-04-28T12:49:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.084+0000 7f4c067fc700 1 -- 172.21.15.53:0/2052523685 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4bfc02a070 con 0x7f4c08103e30 2024-04-28T12:49:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.085+0000 7f4c067fc700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4bf00a3d50 0x7f4bf00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.085+0000 7f4c0d6b6700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4bf00a3d50 0x7f4bf00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.085+0000 7f4c067fc700 1 -- 172.21.15.53:0/2052523685 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f4bfc0d1d00 con 0x7f4c08103e30 2024-04-28T12:49:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.085+0000 7f4c0d6b6700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4bf00a3d50 0x7f4bf00a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8005950 tx=0x7f4bf800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.089+0000 7f4c067fc700 1 -- 172.21.15.53:0/2052523685 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4bfc031070 con 0x7f4c08103e30 2024-04-28T12:49:52.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 -- 172.21.15.53:0/625943966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 msgr2=0x7fee881098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 --2- 172.21.15.53:0/625943966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881098d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fee78009990 tx=0x7fee78009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 -- 172.21.15.53:0/625943966 shutdown_connections 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 --2- 172.21.15.53:0/625943966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881098d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 --2- 172.21.15.53:0/625943966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee88103e90 0x7fee881102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 --2- 172.21.15.53:0/625943966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee881034d0 0x7fee88103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 -- 172.21.15.53:0/625943966 >> 172.21.15.53:0/625943966 conn(0x7fee8807a4b0 msgr2=0x7fee8807a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 -- 172.21.15.53:0/625943966 shutdown_connections 2024-04-28T12:49:52.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.268+0000 7fee8d97a700 1 -- 172.21.15.53:0/625943966 wait complete. 2024-04-28T12:49:52.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.269+0000 7fee8d97a700 1 Processor -- start 2024-04-28T12:49:52.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.269+0000 7fee8d97a700 1 -- start start 2024-04-28T12:49:52.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee8d97a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee881034d0 0x7fee8819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee8d97a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee88103e90 0x7fee8819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee8d97a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee8d97a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee881135f0 con 0x7fee881094f0 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee8d97a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee88113470 con 0x7fee881034d0 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee8d97a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fee88113770 con 0x7fee88103e90 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee877fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee86ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee881034d0 0x7fee8819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee877fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34442/0 (socket says 172.21.15.53:34442) 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee877fe700 1 -- 172.21.15.53:0/655023368 learned_addr learned my addr 172.21.15.53:0/655023368 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee867fc700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee88103e90 0x7fee8819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee86ffd700 1 -- 172.21.15.53:0/655023368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee88103e90 msgr2=0x7fee8819fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee86ffd700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee88103e90 0x7fee8819fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee86ffd700 1 -- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 msgr2=0x7fee881998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee86ffd700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee86ffd700 1 -- 172.21.15.53:0/655023368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8819a050 con 0x7fee881034d0 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.270+0000 7fee877fe700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.271+0000 7fee86ffd700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee881034d0 0x7fee8819f730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee780059d0 tx=0x7fee780047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.271+0000 7fee8c978700 1 -- 172.21.15.53:0/655023368 <== mon.2 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78027070 con 0x7fee881034d0 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.271+0000 7fee8c978700 1 -- 172.21.15.53:0/655023368 <== mon.2 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee78022470 con 0x7fee881034d0 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.271+0000 7fee8c978700 1 -- 172.21.15.53:0/655023368 <== mon.2 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee78017650 con 0x7fee881034d0 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.271+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee8819a2e0 con 0x7fee881034d0 2024-04-28T12:49:52.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.271+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee881a6600 con 0x7fee881034d0 2024-04-28T12:49:52.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.272+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee8806a110 con 0x7fee881034d0 2024-04-28T12:49:52.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.274+0000 7fee8c978700 1 -- 172.21.15.53:0/655023368 <== mon.2 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fee7802a070 con 0x7fee881034d0 2024-04-28T12:49:52.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.274+0000 7fee8c978700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fee700a3cd0 0x7fee700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.274+0000 7fee867fc700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fee700a3cd0 0x7fee700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.274+0000 7fee8c978700 1 -- 172.21.15.53:0/655023368 <== mon.2 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fee780d1b60 con 0x7fee881034d0 2024-04-28T12:49:52.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.275+0000 7fee867fc700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fee700a3cd0 0x7fee700a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fee7c005fd0 tx=0x7fee7c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.278+0000 7fee8c978700 1 -- 172.21.15.53:0/655023368 <== mon.2 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fee78031070 con 0x7fee881034d0 2024-04-28T12:49:52.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.340+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f4c0802e490 con 0x7f4c08103e30 2024-04-28T12:49:52.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.340+0000 7f4c067fc700 1 -- 172.21.15.53:0/2052523685 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4bfc013070 con 0x7f4c08103e30 2024-04-28T12:49:52.337 INFO:teuthology.orchestra.run.smithi053.stdout:55834574925 2024-04-28T12:49:52.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.342+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4bf00a3d50 msgr2=0x7f4bf00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.342+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4bf00a3d50 0x7f4bf00a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8005950 tx=0x7f4bf800a3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.342+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103e30 msgr2=0x7f4c0819fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.342+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103e30 0x7f4c0819fcc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0059d0 tx=0x7f4bfc0047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 shutdown_connections 2024-04-28T12:49:52.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4bf00a3d50 0x7f4bf00a6210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c081094c0 0x7f4c08199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c08103470 0x7f4c0819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 --2- 172.21.15.53:0/2052523685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c08103e30 0x7f4c0819fcc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 >> 172.21.15.53:0/2052523685 conn(0x7f4c0807a580 msgr2=0x7f4c081022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 shutdown_connections 2024-04-28T12:49:52.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.343+0000 7f4c0f91a700 1 -- 172.21.15.53:0/2052523685 wait complete. 2024-04-28T12:49:52.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.491+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fee8802e490 con 0x7fee881034d0 2024-04-28T12:49:52.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.492+0000 7fee8c978700 1 -- 172.21.15.53:0/655023368 <== mon.2 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fee78013070 con 0x7fee881034d0 2024-04-28T12:49:52.489 INFO:teuthology.orchestra.run.smithi053.stdout:292057776166 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fee700a3cd0 msgr2=0x7fee700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fee700a3cd0 0x7fee700a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fee7c005fd0 tx=0x7fee7c005f60 comp rx=0 tx=0).stop 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee881034d0 msgr2=0x7fee8819f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee881034d0 0x7fee8819f730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee780059d0 tx=0x7fee780047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 shutdown_connections 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fee700a3cd0 0x7fee700a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee881094f0 0x7fee881998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee881034d0 0x7fee8819f730 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 --2- 172.21.15.53:0/655023368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee88103e90 0x7fee8819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 >> 172.21.15.53:0/655023368 conn(0x7fee8807a4b0 msgr2=0x7fee88102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 shutdown_connections 2024-04-28T12:49:52.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.494+0000 7fee8d97a700 1 -- 172.21.15.53:0/655023368 wait complete. 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 -- 172.21.15.53:0/2281896885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 msgr2=0x7fb1dc10e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/2281896885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc10e960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400b0a0 tx=0x7fb1d400b3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 -- 172.21.15.53:0/2281896885 shutdown_connections 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/2281896885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc10e960 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/2281896885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1dc107a90 0x7fb1dc107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/2281896885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1dc101ae0 0x7fb1dc101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 -- 172.21.15.53:0/2281896885 >> 172.21.15.53:0/2281896885 conn(0x7fb1dc078990 msgr2=0x7fb1dc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 -- 172.21.15.53:0/2281896885 shutdown_connections 2024-04-28T12:49:52.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.547+0000 7fb1e19c7700 1 -- 172.21.15.53:0/2281896885 wait complete. 2024-04-28T12:49:52.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 Processor -- start 2024-04-28T12:49:52.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 -- start start 2024-04-28T12:49:52.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1dc101ae0 0x7fb1dc19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1dc107a90 0x7fb1dc1a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1124d0 con 0x7fb1dc1024a0 2024-04-28T12:49:52.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc112350 con 0x7fb1dc101ae0 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.548+0000 7fb1e19c7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc112650 con 0x7fb1dc107a90 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc19ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34462/0 (socket says 172.21.15.53:34462) 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1daffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1dc101ae0 0x7fb1dc19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 -- 172.21.15.53:0/1014977555 learned_addr learned my addr 172.21.15.53:0/1014977555 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1db7fe700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1dc107a90 0x7fb1dc1a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 -- 172.21.15.53:0/1014977555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1dc107a90 msgr2=0x7fb1dc1a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1dc107a90 0x7fb1dc1a59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 -- 172.21.15.53:0/1014977555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1dc101ae0 msgr2=0x7fb1dc19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1dc101ae0 0x7fb1dc19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 -- 172.21.15.53:0/1014977555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc1a5ef0 con 0x7fb1dc1024a0 2024-04-28T12:49:52.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1db7fe700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1dc107a90 0x7fb1dc1a59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:52.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1daffd700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1dc101ae0 0x7fb1dc19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:52.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1da7fc700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc19ae70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000e9a0 tx=0x7fb1d000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.549+0000 7fb1e09c5700 1 -- 172.21.15.53:0/1014977555 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000cbd0 con 0x7fb1dc1024a0 2024-04-28T12:49:52.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.550+0000 7fb1e09c5700 1 -- 172.21.15.53:0/1014977555 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1d000cd70 con 0x7fb1dc1024a0 2024-04-28T12:49:52.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.550+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1dc1a60c0 con 0x7fb1dc1024a0 2024-04-28T12:49:52.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.550+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1dc1a6620 con 0x7fb1dc1024a0 2024-04-28T12:49:52.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.550+0000 7fb1e09c5700 1 -- 172.21.15.53:0/1014977555 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0018680 con 0x7fb1dc1024a0 2024-04-28T12:49:52.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.551+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1dc1031d0 con 0x7fb1dc1024a0 2024-04-28T12:49:52.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.552+0000 7fb1e09c5700 1 -- 172.21.15.53:0/1014977555 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb1d0010430 con 0x7fb1dc1024a0 2024-04-28T12:49:52.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.553+0000 7fb1e09c5700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb1c40a3d50 0x7fb1c40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.553+0000 7fb1daffd700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb1c40a3d50 0x7fb1c40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.554+0000 7fb1e09c5700 1 -- 172.21.15.53:0/1014977555 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fb1d00d0c50 con 0x7fb1dc1024a0 2024-04-28T12:49:52.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.554+0000 7fb1daffd700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb1c40a3d50 0x7fb1c40a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc007f50 tx=0x7fb1cc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.557+0000 7fb1e09c5700 1 -- 172.21.15.53:0/1014977555 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb1d009d490 con 0x7fb1dc1024a0 2024-04-28T12:49:52.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.770+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fb1dc02e490 con 0x7fb1dc1024a0 2024-04-28T12:49:52.767 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:52 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2052523685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T12:49:52.767 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:52 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/655023368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T12:49:52.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.770+0000 7fb1e09c5700 1 -- 172.21.15.53:0/1014977555 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fb1d009cd90 con 0x7fb1dc1024a0 2024-04-28T12:49:52.767 INFO:teuthology.orchestra.run.smithi053.stdout:313532612639 2024-04-28T12:49:52.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb1c40a3d50 msgr2=0x7fb1c40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb1c40a3d50 0x7fb1c40a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc007f50 tx=0x7fb1cc0073d0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 msgr2=0x7fb1dc19ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc19ae70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000e9a0 tx=0x7fb1d000ecb0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 shutdown_connections 2024-04-28T12:49:52.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb1c40a3d50 0x7fb1c40a6210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1dc107a90 0x7fb1dc1a59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1dc101ae0 0x7fb1dc19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 --2- 172.21.15.53:0/1014977555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1dc1024a0 0x7fb1dc19ae70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 >> 172.21.15.53:0/1014977555 conn(0x7fb1dc078990 msgr2=0x7fb1dc104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 shutdown_connections 2024-04-28T12:49:52.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:52.773+0000 7fb1e19c7700 1 -- 172.21.15.53:0/1014977555 wait complete. 2024-04-28T12:49:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:52 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2052523685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T12:49:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:52 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/655023368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T12:49:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:52 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2052523685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T12:49:52.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:52 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/655023368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T12:49:53.038 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299434 got 223338299439 for osd.7 2024-04-28T12:49:53.038 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:53.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 -- 172.21.15.53:0/69891936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 msgr2=0x7f059c1102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:53.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/69891936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c1102d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f058c009990 tx=0x7f058c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 -- 172.21.15.53:0/69891936 shutdown_connections 2024-04-28T12:49:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/69891936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c1102d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/69891936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f059c109490 0x7f059c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/69891936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c06b9c0 0x7f059c06be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 -- 172.21.15.53:0/69891936 >> 172.21.15.53:0/69891936 conn(0x7f059c079290 msgr2=0x7f059c0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 -- 172.21.15.53:0/69891936 shutdown_connections 2024-04-28T12:49:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.471+0000 7f05a1b7d700 1 -- 172.21.15.53:0/69891936 wait complete. 2024-04-28T12:49:53.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 Processor -- start 2024-04-28T12:49:53.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 -- start start 2024-04-28T12:49:53.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c06b9c0 0x7f059c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:53.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:53.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f059c109490 0x7f059c199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f059c1135e0 con 0x7f059c06c360 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f059c113460 con 0x7f059c109490 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f05a1b7d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f059c113760 con 0x7f059c06b9c0 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f059affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f059affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c19fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34478/0 (socket says 172.21.15.53:34478) 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f059affd700 1 -- 172.21.15.53:0/2762700585 learned_addr learned my addr 172.21.15.53:0/2762700585 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.472+0000 7f059b7fe700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c06b9c0 0x7f059c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059bfff700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f059c109490 0x7f059c199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059affd700 1 -- 172.21.15.53:0/2762700585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c06b9c0 msgr2=0x7f059c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059affd700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c06b9c0 0x7f059c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059affd700 1 -- 172.21.15.53:0/2762700585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f059c109490 msgr2=0x7f059c199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059affd700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f059c109490 0x7f059c199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059affd700 1 -- 172.21.15.53:0/2762700585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059c19a0e0 con 0x7f059c06c360 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059bfff700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f059c109490 0x7f059c199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f059affd700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c19fcb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f058400d950 tx=0x7f058400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f0598ff9700 1 -- 172.21.15.53:0/2762700585 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584016070 con 0x7f059c06c360 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059c19a3d0 con 0x7f059c06c360 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f0598ff9700 1 -- 172.21.15.53:0/2762700585 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0584009df0 con 0x7f059c06c360 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f0598ff9700 1 -- 172.21.15.53:0/2762700585 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058400f3c0 con 0x7f059c06c360 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.473+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f059c1a6640 con 0x7f059c06c360 2024-04-28T12:49:53.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.474+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059c06a110 con 0x7f059c06c360 2024-04-28T12:49:53.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.475+0000 7f0598ff9700 1 -- 172.21.15.53:0/2762700585 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0584019070 con 0x7f059c06c360 2024-04-28T12:49:53.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.476+0000 7f0598ff9700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3cd0 0x7f05880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:53.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.476+0000 7f059b7fe700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3cd0 0x7f05880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:53.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.476+0000 7f0598ff9700 1 -- 172.21.15.53:0/2762700585 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f05840d0160 con 0x7f059c06c360 2024-04-28T12:49:53.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.476+0000 7f059b7fe700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3cd0 0x7f05880a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0594009550 tx=0x7f05940091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:53.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.480+0000 7f0598ff9700 1 -- 172.21.15.53:0/2762700585 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f058409c9e0 con 0x7f059c06c360 2024-04-28T12:49:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:53 smithi053 ceph-mon[29182]: pgmap v327: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:53.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:53 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1014977555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T12:49:53.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.690+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f059c02e490 con 0x7f059c06c360 2024-04-28T12:49:53.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.691+0000 7f0598ff9700 1 -- 172.21.15.53:0/2762700585 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f058409c2e0 con 0x7f059c06c360 2024-04-28T12:49:53.687 INFO:teuthology.orchestra.run.smithi053.stdout:146028888128 2024-04-28T12:49:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3cd0 msgr2=0x7f05880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3cd0 0x7f05880a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0594009550 tx=0x7f05940091b0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 msgr2=0x7f059c19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c19fcb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f058400d950 tx=0x7f058400dc60 comp rx=0 tx=0).stop 2024-04-28T12:49:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 shutdown_connections 2024-04-28T12:49:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3cd0 0x7f05880a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f059c109490 0x7f059c199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c06b9c0 0x7f059c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 --2- 172.21.15.53:0/2762700585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c06c360 0x7f059c19fcb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:53.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 >> 172.21.15.53:0/2762700585 conn(0x7f059c079290 msgr2=0x7f059c100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:53.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 shutdown_connections 2024-04-28T12:49:53.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:53.693+0000 7f05a1b7d700 1 -- 172.21.15.53:0/2762700585 wait complete. 2024-04-28T12:49:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:53 smithi191 ceph-mon[33016]: pgmap v327: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:53 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1014977555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T12:49:53.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:53 smithi077 ceph-mon[32553]: pgmap v327: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:53.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:53 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1014977555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T12:49:54.565 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574917 got 55834574925 for osd.1 2024-04-28T12:49:54.565 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:54 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2762700585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T12:49:54.854 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:54 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2762700585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T12:49:54.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:54 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2762700585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T12:49:54.951 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776162 got 292057776166 for osd.9 2024-04-28T12:49:54.951 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:55.423 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149759 got 111669149764 for osd.3 2024-04-28T12:49:55.423 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:55.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:55 smithi053 ceph-mon[29182]: pgmap v328: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:55.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:55 smithi077 ceph-mon[32553]: pgmap v328: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:56.018 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612634 got 313532612639 for osd.10 2024-04-28T12:49:56.018 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:56.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:55 smithi191 ceph-mon[33016]: pgmap v328: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:49:56.446 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411400 for osd.2 2024-04-28T12:49:56.447 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:56.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 -- 172.21.15.53:0/3498186600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d41034a0 msgr2=0x7fa0d4103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:56.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3498186600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d41034a0 0x7fa0d4103920 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009990 tx=0x7fa0c0009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 -- 172.21.15.53:0/3498186600 shutdown_connections 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3498186600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0d4103e60 0x7fa0d41102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3498186600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d41094e0 0x7fa0d41098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3498186600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d41034a0 0x7fa0d4103920 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 -- 172.21.15.53:0/3498186600 >> 172.21.15.53:0/3498186600 conn(0x7fa0d407a590 msgr2=0x7fa0d407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 -- 172.21.15.53:0/3498186600 shutdown_connections 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.774+0000 7fa0dac19700 1 -- 172.21.15.53:0/3498186600 wait complete. 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 Processor -- start 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 -- start start 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0d41034a0 0x7fa0d419f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d4103e60 0x7fa0d419fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d41094e0 0x7fa0d4199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d41136e0 con 0x7fa0d4103e60 2024-04-28T12:49:56.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4113560 con 0x7fa0d41094e0 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0dac19700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4113860 con 0x7fa0d41034a0 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d4103e60 0x7fa0d419fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d4103e60 0x7fa0d419fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43666/0 (socket says 172.21.15.53:43666) 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d3fff700 1 -- 172.21.15.53:0/3558955915 learned_addr learned my addr 172.21.15.53:0/3558955915 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d89b5700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0d41034a0 0x7fa0d419f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d3fff700 1 -- 172.21.15.53:0/3558955915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0d41034a0 msgr2=0x7fa0d419f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d3fff700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0d41034a0 0x7fa0d419f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d3fff700 1 -- 172.21.15.53:0/3558955915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d41094e0 msgr2=0x7fa0d4199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d91b6700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d41094e0 0x7fa0d4199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.775+0000 7fa0d3fff700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d41094e0 0x7fa0d4199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0d3fff700 1 -- 172.21.15.53:0/3558955915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d419a1c0 con 0x7fa0d4103e60 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0d91b6700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d41094e0 0x7fa0d4199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0d3fff700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d4103e60 0x7fa0d419fde0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0000c00 tx=0x7fa0c00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:56.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0d1ffb700 1 -- 172.21.15.53:0/3558955915 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0027070 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d419a450 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0d1ffb700 1 -- 172.21.15.53:0/3558955915 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c00041d0 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0d1ffb700 1 -- 172.21.15.53:0/3558955915 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c000f460 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.776+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d41a6770 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.777+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d406a110 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.778+0000 7fa0d1ffb700 1 -- 172.21.15.53:0/3558955915 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa0c002a070 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.779+0000 7fa0d1ffb700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0c40a3cd0 0x7fa0c40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.779+0000 7fa0d89b5700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0c40a3cd0 0x7fa0c40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.779+0000 7fa0d1ffb700 1 -- 172.21.15.53:0/3558955915 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fa0c00d1570 con 0x7fa0d4103e60 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.779+0000 7fa0d89b5700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0c40a3cd0 0x7fa0c40a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8006fd0 tx=0x7fa0c8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:56.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.783+0000 7fa0d1ffb700 1 -- 172.21.15.53:0/3558955915 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa0c005f060 con 0x7fa0d4103e60 2024-04-28T12:49:56.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.994+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fa0d402e490 con 0x7fa0d4103e60 2024-04-28T12:49:56.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.994+0000 7fa0d1ffb700 1 -- 172.21.15.53:0/3558955915 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa0c0013070 con 0x7fa0d4103e60 2024-04-28T12:49:56.990 INFO:teuthology.orchestra.run.smithi053.stdout:257698037804 2024-04-28T12:49:56.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0c40a3cd0 msgr2=0x7fa0c40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:56.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0c40a3cd0 0x7fa0c40a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8006fd0 tx=0x7fa0c8008040 comp rx=0 tx=0).stop 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d4103e60 msgr2=0x7fa0d419fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d4103e60 0x7fa0d419fde0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0000c00 tx=0x7fa0c00047b0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 shutdown_connections 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa0c40a3cd0 0x7fa0c40a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d41094e0 0x7fa0d4199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0d41034a0 0x7fa0d419f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 --2- 172.21.15.53:0/3558955915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0d4103e60 0x7fa0d419fde0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 >> 172.21.15.53:0/3558955915 conn(0x7fa0d407a590 msgr2=0x7fa0d4102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 shutdown_connections 2024-04-28T12:49:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:49:56.996+0000 7fa0dac19700 1 -- 172.21.15.53:0/3558955915 wait complete. 2024-04-28T12:49:57.256 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888122 got 146028888128 for osd.4 2024-04-28T12:49:57.256 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:57.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:57 smithi053 ceph-mon[29182]: pgmap v329: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:57.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:57 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3558955915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T12:49:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:57 smithi077 ceph-mon[32553]: pgmap v329: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:57.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:57 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3558955915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T12:49:58.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037799 got 257698037804 for osd.8 2024-04-28T12:49:58.003 DEBUG:teuthology.parallel:result is None 2024-04-28T12:49:58.003 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-28T12:49:58.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-28T12:49:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:57 smithi191 ceph-mon[33016]: pgmap v329: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:49:58.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:57 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3558955915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T12:49:58.774 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:59.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:49:59 smithi191 ceph-mon[33016]: pgmap v330: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:59.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:49:59 smithi053 ceph-mon[29182]: pgmap v330: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:49:59.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:49:59 smithi077 ceph-mon[32553]: pgmap v330: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:00.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1450789579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006b130 msgr2=0x7fb7e006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1450789579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006b130 0x7fb7e006b590 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009990 tx=0x7fb7d0009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1450789579 shutdown_connections 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1450789579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7e006bad0 0x7fb7e01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1450789579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e01094a0 0x7fb7e0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1450789579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006b130 0x7fb7e006b590 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1450789579 >> 172.21.15.53:0/1450789579 conn(0x7fb7e00792d0 msgr2=0x7fb7e00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1450789579 shutdown_connections 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.484+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1450789579 wait complete. 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 Processor -- start 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 -- start start 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7e006b130 0x7fb7e0199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006bad0 0x7fb7e019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e01094a0 0x7fb7e019e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e0113650 con 0x7fb7e006bad0 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01134d0 con 0x7fb7e01094a0 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e79f7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01137d0 con 0x7fb7e006b130 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e4f92700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006bad0 0x7fb7e019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e4f92700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006bad0 0x7fb7e019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43680/0 (socket says 172.21.15.53:43680) 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e4f92700 1 -- 172.21.15.53:0/1876021560 learned_addr learned my addr 172.21.15.53:0/1876021560 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e5793700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7e006b130 0x7fb7e0199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.485+0000 7fb7e5f94700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e01094a0 0x7fb7e019e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e4f92700 1 -- 172.21.15.53:0/1876021560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7e006b130 msgr2=0x7fb7e0199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e4f92700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7e006b130 0x7fb7e0199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e4f92700 1 -- 172.21.15.53:0/1876021560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e01094a0 msgr2=0x7fb7e019e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e4f92700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e01094a0 0x7fb7e019e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e4f92700 1 -- 172.21.15.53:0/1876021560 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e019eb80 con 0x7fb7e006bad0 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e5f94700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e01094a0 0x7fb7e019e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:00.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e4f92700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006bad0 0x7fb7e019a0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0000c00 tx=0x7fb7d00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:00.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7d67fc700 1 -- 172.21.15.53:0/1876021560 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0027070 con 0x7fb7e006bad0 2024-04-28T12:50:00.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e019ee10 con 0x7fb7e006bad0 2024-04-28T12:50:00.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7d67fc700 1 -- 172.21.15.53:0/1876021560 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7d00040e0 con 0x7fb7e006bad0 2024-04-28T12:50:00.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7d67fc700 1 -- 172.21.15.53:0/1876021560 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d000f460 con 0x7fb7e006bad0 2024-04-28T12:50:00.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.486+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e01a6730 con 0x7fb7e006bad0 2024-04-28T12:50:00.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.487+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e0104b50 con 0x7fb7e006bad0 2024-04-28T12:50:00.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.489+0000 7fb7d67fc700 1 -- 172.21.15.53:0/1876021560 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb7d002a070 con 0x7fb7e006bad0 2024-04-28T12:50:00.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.490+0000 7fb7d67fc700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb7cc0a3cd0 0x7fb7cc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.490+0000 7fb7e5793700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb7cc0a3cd0 0x7fb7cc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.490+0000 7fb7d67fc700 1 -- 172.21.15.53:0/1876021560 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fb7d00d19e0 con 0x7fb7e006bad0 2024-04-28T12:50:00.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.490+0000 7fb7e5793700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb7cc0a3cd0 0x7fb7cc0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc005950 tx=0x7fb7dc00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:00.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.494+0000 7fb7d67fc700 1 -- 172.21.15.53:0/1876021560 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb7d0031070 con 0x7fb7e006bad0 2024-04-28T12:50:00.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.724+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7e019aee0 con 0x7fb7cc0a3cd0 2024-04-28T12:50:00.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.733+0000 7fb7d67fc700 1 -- 172.21.15.53:0/1876021560 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259021 (secure 0 0 0) 0x7fb7e019aee0 con 0x7fb7cc0a3cd0 2024-04-28T12:50:00.733 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb7cc0a3cd0 msgr2=0x7fb7cc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb7cc0a3cd0 0x7fb7cc0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc005950 tx=0x7fb7dc00a300 comp rx=0 tx=0).stop 2024-04-28T12:50:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006bad0 msgr2=0x7fb7e019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006bad0 0x7fb7e019a0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0000c00 tx=0x7fb7d00047b0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 shutdown_connections 2024-04-28T12:50:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fb7cc0a3cd0 0x7fb7cc0a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7e01094a0 0x7fb7e019e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7e006b130 0x7fb7e0199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 --2- 172.21.15.53:0/1876021560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7e006bad0 0x7fb7e019a0c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 >> 172.21.15.53:0/1876021560 conn(0x7fb7e00792d0 msgr2=0x7fb7e0100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:00.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 shutdown_connections 2024-04-28T12:50:00.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:00.736+0000 7fb7e79f7700 1 -- 172.21.15.53:0/1876021560 wait complete. 2024-04-28T12:50:00.736 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-04-28T12:50:00.744 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:00 smithi053 ceph-mon[29182]: overall HEALTH_OK 2024-04-28T12:50:00.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:00 smithi077 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T12:50:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:00 smithi191 ceph-mon[33016]: overall HEALTH_OK 2024-04-28T12:50:01.451 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-04-28T12:50:00.469790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644610,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":554,"num_read_kb":380,"num_write":335,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1231156,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377460,"kb_used_data":11596,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564364,"statfs":{"total":1151940427776,"available":1151553908736,"internally_reserved":0,"allocated":11874304,"data_stored":7438866,"data_compressed":189761,"data_compressed_allocated":253952,"data_compressed_original":2031616,"omap_allocated":19119,"internal_metadata":374453585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003354"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630828+0000","last_change":"2024-04-28T12:48:44.486823+0000","last_active":"2024-04-28T12:49:35.630828+0000","last_peered":"2024-04-28T12:49:35.630828+0000","last_clean":"2024-04-28T12:49:35.630828+0000","last_became_active":"2024-04-28T12:48:44.485159+0000","last_became_peered":"2024-04-28T12:48:44.485159+0000","last_unstale":"2024-04-28T12:49:35.630828+0000","last_undegraded":"2024-04-28T12:49:35.630828+0000","last_fullsized":"2024-04-28T12:49:35.630828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:12.126908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630923+0000","last_change":"2024-04-28T12:48:44.477801+0000","last_active":"2024-04-28T12:49:35.630923+0000","last_peered":"2024-04-28T12:49:35.630923+0000","last_clean":"2024-04-28T12:49:35.630923+0000","last_became_active":"2024-04-28T12:48:44.477448+0000","last_became_peered":"2024-04-28T12:48:44.477448+0000","last_unstale":"2024-04-28T12:49:35.630923+0000","last_undegraded":"2024-04-28T12:49:35.630923+0000","last_fullsized":"2024-04-28T12:49:35.630923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:30.350306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630895+0000","last_change":"2024-04-28T12:48:44.485638+0000","last_active":"2024-04-28T12:49:35.630895+0000","last_peered":"2024-04-28T12:49:35.630895+0000","last_clean":"2024-04-28T12:49:35.630895+0000","last_became_active":"2024-04-28T12:48:44.485247+0000","last_became_peered":"2024-04-28T12:48:44.485247+0000","last_unstale":"2024-04-28T12:49:35.630895+0000","last_undegraded":"2024-04-28T12:49:35.630895+0000","last_fullsized":"2024-04-28T12:49:35.630895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:49:35.209786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075121+0000","last_change":"2024-04-28T12:48:44.484329+0000","last_active":"2024-04-28T12:49:36.075121+0000","last_peered":"2024-04-28T12:49:36.075121+0000","last_clean":"2024-04-28T12:49:36.075121+0000","last_became_active":"2024-04-28T12:48:44.483980+0000","last_became_peered":"2024-04-28T12:48:44.483980+0000","last_unstale":"2024-04-28T12:49:36.075121+0000","last_undegraded":"2024-04-28T12:49:36.075121+0000","last_fullsized":"2024-04-28T12:49:36.075121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:14:47.570749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635150+0000","last_change":"2024-04-28T12:48:44.487609+0000","last_active":"2024-04-28T12:49:35.635150+0000","last_peered":"2024-04-28T12:49:35.635150+0000","last_clean":"2024-04-28T12:49:35.635150+0000","last_became_active":"2024-04-28T12:48:44.487199+0000","last_became_peered":"2024-04-28T12:48:44.487199+0000","last_unstale":"2024-04-28T12:49:35.635150+0000","last_undegraded":"2024-04-28T12:49:35.635150+0000","last_fullsized":"2024-04-28T12:49:35.635150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:30:03.675503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634772+0000","last_change":"2024-04-28T12:48:44.477889+0000","last_active":"2024-04-28T12:49:35.634772+0000","last_peered":"2024-04-28T12:49:35.634772+0000","last_clean":"2024-04-28T12:49:35.634772+0000","last_became_active":"2024-04-28T12:48:44.477472+0000","last_became_peered":"2024-04-28T12:48:44.477472+0000","last_unstale":"2024-04-28T12:49:35.634772+0000","last_undegraded":"2024-04-28T12:49:35.634772+0000","last_fullsized":"2024-04-28T12:49:35.634772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:35:03.684380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631190+0000","last_change":"2024-04-28T12:48:44.482889+0000","last_active":"2024-04-28T12:49:35.631190+0000","last_peered":"2024-04-28T12:49:35.631190+0000","last_clean":"2024-04-28T12:49:35.631190+0000","last_became_active":"2024-04-28T12:48:44.482569+0000","last_became_peered":"2024-04-28T12:48:44.482569+0000","last_unstale":"2024-04-28T12:49:35.631190+0000","last_undegraded":"2024-04-28T12:49:35.631190+0000","last_fullsized":"2024-04-28T12:49:35.631190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:07.858792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810695+0000","last_change":"2024-04-28T12:48:44.484118+0000","last_active":"2024-04-28T12:49:35.810695+0000","last_peered":"2024-04-28T12:49:35.810695+0000","last_clean":"2024-04-28T12:49:35.810695+0000","last_became_active":"2024-04-28T12:48:44.483577+0000","last_became_peered":"2024-04-28T12:48:44.483577+0000","last_unstale":"2024-04-28T12:49:35.810695+0000","last_undegraded":"2024-04-28T12:49:35.810695+0000","last_fullsized":"2024-04-28T12:49:35.810695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:52:24.574445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.639014+0000","last_change":"2024-04-28T12:48:44.485372+0000","last_active":"2024-04-28T12:49:35.639014+0000","last_peered":"2024-04-28T12:49:35.639014+0000","last_clean":"2024-04-28T12:49:35.639014+0000","last_became_active":"2024-04-28T12:48:44.485074+0000","last_became_peered":"2024-04-28T12:48:44.485074+0000","last_unstale":"2024-04-28T12:49:35.639014+0000","last_undegraded":"2024-04-28T12:49:35.639014+0000","last_fullsized":"2024-04-28T12:49:35.639014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:24:39.390059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638394+0000","last_change":"2024-04-28T12:49:35.638394+0000","last_active":"2024-04-28T12:49:35.638394+0000","last_peered":"2024-04-28T12:49:35.638394+0000","last_clean":"2024-04-28T12:49:35.638394+0000","last_became_active":"2024-04-28T12:49:35.637821+0000","last_became_peered":"2024-04-28T12:49:35.637821+0000","last_unstale":"2024-04-28T12:49:35.638394+0000","last_undegraded":"2024-04-28T12:49:35.638394+0000","last_fullsized":"2024-04-28T12:49:35.638394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:10:54.535365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631517+0000","last_change":"2024-04-28T12:48:44.479273+0000","last_active":"2024-04-28T12:49:35.631517+0000","last_peered":"2024-04-28T12:49:35.631517+0000","last_clean":"2024-04-28T12:49:35.631517+0000","last_became_active":"2024-04-28T12:48:44.478962+0000","last_became_peered":"2024-04-28T12:48:44.478962+0000","last_unstale":"2024-04-28T12:49:35.631517+0000","last_undegraded":"2024-04-28T12:49:35.631517+0000","last_fullsized":"2024-04-28T12:49:35.631517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:27.698365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074991+0000","last_change":"2024-04-28T12:48:44.478442+0000","last_active":"2024-04-28T12:49:36.074991+0000","last_peered":"2024-04-28T12:49:36.074991+0000","last_clean":"2024-04-28T12:49:36.074991+0000","last_became_active":"2024-04-28T12:48:44.478121+0000","last_became_peered":"2024-04-28T12:48:44.478121+0000","last_unstale":"2024-04-28T12:49:36.074991+0000","last_undegraded":"2024-04-28T12:49:36.074991+0000","last_fullsized":"2024-04-28T12:49:36.074991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:07:43.195804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074409+0000","last_change":"2024-04-28T12:48:44.476475+0000","last_active":"2024-04-28T12:49:36.074409+0000","last_peered":"2024-04-28T12:49:36.074409+0000","last_clean":"2024-04-28T12:49:36.074409+0000","last_became_active":"2024-04-28T12:48:44.475839+0000","last_became_peered":"2024-04-28T12:48:44.475839+0000","last_unstale":"2024-04-28T12:49:36.074409+0000","last_undegraded":"2024-04-28T12:49:36.074409+0000","last_fullsized":"2024-04-28T12:49:36.074409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:34:51.513356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630656+0000","last_change":"2024-04-28T12:48:44.480569+0000","last_active":"2024-04-28T12:49:35.630656+0000","last_peered":"2024-04-28T12:49:35.630656+0000","last_clean":"2024-04-28T12:49:35.630656+0000","last_became_active":"2024-04-28T12:48:44.478085+0000","last_became_peered":"2024-04-28T12:48:44.478085+0000","last_unstale":"2024-04-28T12:49:35.630656+0000","last_undegraded":"2024-04-28T12:49:35.630656+0000","last_fullsized":"2024-04-28T12:49:35.630656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:23:13.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.636591+0000","last_change":"2024-04-28T12:49:35.636591+0000","last_active":"2024-04-28T12:49:35.636591+0000","last_peered":"2024-04-28T12:49:35.636591+0000","last_clean":"2024-04-28T12:49:35.636591+0000","last_became_active":"2024-04-28T12:49:35.636212+0000","last_became_peered":"2024-04-28T12:49:35.636212+0000","last_unstale":"2024-04-28T12:49:35.636591+0000","last_undegraded":"2024-04-28T12:49:35.636591+0000","last_fullsized":"2024-04-28T12:49:35.636591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:49:13.868018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638020+0000","last_change":"2024-04-28T12:48:44.485600+0000","last_active":"2024-04-28T12:49:35.638020+0000","last_peered":"2024-04-28T12:49:35.638020+0000","last_clean":"2024-04-28T12:49:35.638020+0000","last_became_active":"2024-04-28T12:48:44.485297+0000","last_became_peered":"2024-04-28T12:48:44.485297+0000","last_unstale":"2024-04-28T12:49:35.638020+0000","last_undegraded":"2024-04-28T12:49:35.638020+0000","last_fullsized":"2024-04-28T12:49:35.638020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:25:16.124798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630967+0000","last_change":"2024-04-28T12:48:44.487376+0000","last_active":"2024-04-28T12:49:35.630967+0000","last_peered":"2024-04-28T12:49:35.630967+0000","last_clean":"2024-04-28T12:49:35.630967+0000","last_became_active":"2024-04-28T12:48:44.486561+0000","last_became_peered":"2024-04-28T12:48:44.486561+0000","last_unstale":"2024-04-28T12:49:35.630967+0000","last_undegraded":"2024-04-28T12:49:35.630967+0000","last_fullsized":"2024-04-28T12:49:35.630967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:43.398130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628490+0000","last_change":"2024-04-28T12:48:44.475470+0000","last_active":"2024-04-28T12:49:35.628490+0000","last_peered":"2024-04-28T12:49:35.628490+0000","last_clean":"2024-04-28T12:49:35.628490+0000","last_became_active":"2024-04-28T12:48:44.475113+0000","last_became_peered":"2024-04-28T12:48:44.475113+0000","last_unstale":"2024-04-28T12:49:35.628490+0000","last_undegraded":"2024-04-28T12:49:35.628490+0000","last_fullsized":"2024-04-28T12:49:35.628490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:55:28.688458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634384+0000","last_change":"2024-04-28T12:48:44.482195+0000","last_active":"2024-04-28T12:49:35.634384+0000","last_peered":"2024-04-28T12:49:35.634384+0000","last_clean":"2024-04-28T12:49:35.634384+0000","last_became_active":"2024-04-28T12:48:44.481847+0000","last_became_peered":"2024-04-28T12:48:44.481847+0000","last_unstale":"2024-04-28T12:49:35.634384+0000","last_undegraded":"2024-04-28T12:49:35.634384+0000","last_fullsized":"2024-04-28T12:49:35.634384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:18:03.507012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628309+0000","last_change":"2024-04-28T12:48:43.517808+0000","last_active":"2024-04-28T12:49:35.628309+0000","last_peered":"2024-04-28T12:49:35.628309+0000","last_clean":"2024-04-28T12:49:35.628309+0000","last_became_active":"2024-04-28T12:48:43.517443+0000","last_became_peered":"2024-04-28T12:48:43.517443+0000","last_unstale":"2024-04-28T12:49:35.628309+0000","last_undegraded":"2024-04-28T12:49:35.628309+0000","last_fullsized":"2024-04-28T12:49:35.628309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:57:11.972794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811121+0000","last_change":"2024-04-28T12:48:44.486296+0000","last_active":"2024-04-28T12:49:35.811121+0000","last_peered":"2024-04-28T12:49:35.811121+0000","last_clean":"2024-04-28T12:49:35.811121+0000","last_became_active":"2024-04-28T12:48:44.485289+0000","last_became_peered":"2024-04-28T12:48:44.485289+0000","last_unstale":"2024-04-28T12:49:35.811121+0000","last_undegraded":"2024-04-28T12:49:35.811121+0000","last_fullsized":"2024-04-28T12:49:35.811121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:07.558478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811400+0000","last_change":"2024-04-28T12:48:43.523725+0000","last_active":"2024-04-28T12:49:35.811400+0000","last_peered":"2024-04-28T12:49:35.811400+0000","last_clean":"2024-04-28T12:49:35.811400+0000","last_became_active":"2024-04-28T12:48:43.523367+0000","last_became_peered":"2024-04-28T12:48:43.523367+0000","last_unstale":"2024-04-28T12:49:35.811400+0000","last_undegraded":"2024-04-28T12:49:35.811400+0000","last_fullsized":"2024-04-28T12:49:35.811400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:15:15.674552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631210+0000","last_change":"2024-04-28T12:48:44.473393+0000","last_active":"2024-04-28T12:49:35.631210+0000","last_peered":"2024-04-28T12:49:35.631210+0000","last_clean":"2024-04-28T12:49:35.631210+0000","last_became_active":"2024-04-28T12:48:44.473035+0000","last_became_peered":"2024-04-28T12:48:44.473035+0000","last_unstale":"2024-04-28T12:49:35.631210+0000","last_undegraded":"2024-04-28T12:49:35.631210+0000","last_fullsized":"2024-04-28T12:49:35.631210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:06.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.647559+0000","last_change":"2024-04-28T12:49:35.647105+0000","last_active":"2024-04-28T12:49:35.647559+0000","last_peered":"2024-04-28T12:49:35.647559+0000","last_clean":"2024-04-28T12:49:35.647559+0000","last_became_active":"2024-04-28T12:49:35.636538+0000","last_became_peered":"2024-04-28T12:49:35.636538+0000","last_unstale":"2024-04-28T12:49:35.647559+0000","last_undegraded":"2024-04-28T12:49:35.647559+0000","last_fullsized":"2024-04-28T12:49:35.647559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:33:20.769651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638947+0000","last_change":"2024-04-28T12:48:44.475616+0000","last_active":"2024-04-28T12:49:35.638947+0000","last_peered":"2024-04-28T12:49:35.638947+0000","last_clean":"2024-04-28T12:49:35.638947+0000","last_became_active":"2024-04-28T12:48:44.475268+0000","last_became_peered":"2024-04-28T12:48:44.475268+0000","last_unstale":"2024-04-28T12:49:35.638947+0000","last_undegraded":"2024-04-28T12:49:35.638947+0000","last_fullsized":"2024-04-28T12:49:35.638947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:15:25.896749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635723+0000","last_change":"2024-04-28T12:48:43.517652+0000","last_active":"2024-04-28T12:49:35.635723+0000","last_peered":"2024-04-28T12:49:35.635723+0000","last_clean":"2024-04-28T12:49:35.635723+0000","last_became_active":"2024-04-28T12:48:43.517220+0000","last_became_peered":"2024-04-28T12:48:43.517220+0000","last_unstale":"2024-04-28T12:49:35.635723+0000","last_undegraded":"2024-04-28T12:49:35.635723+0000","last_fullsized":"2024-04-28T12:49:35.635723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:25:52.793542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075072+0000","last_change":"2024-04-28T12:48:44.488860+0000","last_active":"2024-04-28T12:49:36.075072+0000","last_peered":"2024-04-28T12:49:36.075072+0000","last_clean":"2024-04-28T12:49:36.075072+0000","last_became_active":"2024-04-28T12:48:44.488487+0000","last_became_peered":"2024-04-28T12:48:44.488487+0000","last_unstale":"2024-04-28T12:49:36.075072+0000","last_undegraded":"2024-04-28T12:49:36.075072+0000","last_fullsized":"2024-04-28T12:49:36.075072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:33:28.357280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:57.558809+0000","last_change":"2024-04-28T12:48:43.507961+0000","last_active":"2024-04-28T12:49:57.558809+0000","last_peered":"2024-04-28T12:49:57.558809+0000","last_clean":"2024-04-28T12:49:57.558809+0000","last_became_active":"2024-04-28T12:48:43.507600+0000","last_became_peered":"2024-04-28T12:48:43.507600+0000","last_unstale":"2024-04-28T12:49:57.558809+0000","last_undegraded":"2024-04-28T12:49:57.558809+0000","last_fullsized":"2024-04-28T12:49:57.558809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:16.188154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634653+0000","last_change":"2024-04-28T12:48:44.474027+0000","last_active":"2024-04-28T12:49:35.634653+0000","last_peered":"2024-04-28T12:49:35.634653+0000","last_clean":"2024-04-28T12:49:35.634653+0000","last_became_active":"2024-04-28T12:48:44.473372+0000","last_became_peered":"2024-04-28T12:48:44.473372+0000","last_unstale":"2024-04-28T12:49:35.634653+0000","last_undegraded":"2024-04-28T12:49:35.634653+0000","last_fullsized":"2024-04-28T12:49:35.634653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:18:43.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634657+0000","last_change":"2024-04-28T12:48:43.521135+0000","last_active":"2024-04-28T12:49:35.634657+0000","last_peered":"2024-04-28T12:49:35.634657+0000","last_clean":"2024-04-28T12:49:35.634657+0000","last_became_active":"2024-04-28T12:48:43.520749+0000","last_became_peered":"2024-04-28T12:48:43.520749+0000","last_unstale":"2024-04-28T12:49:35.634657+0000","last_undegraded":"2024-04-28T12:49:35.634657+0000","last_fullsized":"2024-04-28T12:49:35.634657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:35:33.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074946+0000","last_change":"2024-04-28T12:48:44.479090+0000","last_active":"2024-04-28T12:49:36.074946+0000","last_peered":"2024-04-28T12:49:36.074946+0000","last_clean":"2024-04-28T12:49:36.074946+0000","last_became_active":"2024-04-28T12:48:44.478725+0000","last_became_peered":"2024-04-28T12:48:44.478725+0000","last_unstale":"2024-04-28T12:49:36.074946+0000","last_undegraded":"2024-04-28T12:49:36.074946+0000","last_fullsized":"2024-04-28T12:49:36.074946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:40:21.710990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074274+0000","last_change":"2024-04-28T12:48:43.505309+0000","last_active":"2024-04-28T12:49:36.074274+0000","last_peered":"2024-04-28T12:49:36.074274+0000","last_clean":"2024-04-28T12:49:36.074274+0000","last_became_active":"2024-04-28T12:48:43.504934+0000","last_became_peered":"2024-04-28T12:48:43.504934+0000","last_unstale":"2024-04-28T12:49:36.074274+0000","last_undegraded":"2024-04-28T12:49:36.074274+0000","last_fullsized":"2024-04-28T12:49:36.074274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:48:39.652606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810767+0000","last_change":"2024-04-28T12:48:44.479930+0000","last_active":"2024-04-28T12:49:35.810767+0000","last_peered":"2024-04-28T12:49:35.810767+0000","last_clean":"2024-04-28T12:49:35.810767+0000","last_became_active":"2024-04-28T12:48:44.479571+0000","last_became_peered":"2024-04-28T12:48:44.479571+0000","last_unstale":"2024-04-28T12:49:35.810767+0000","last_undegraded":"2024-04-28T12:49:35.810767+0000","last_fullsized":"2024-04-28T12:49:35.810767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:00.130664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074739+0000","last_change":"2024-04-28T12:48:43.508935+0000","last_active":"2024-04-28T12:49:36.074739+0000","last_peered":"2024-04-28T12:49:36.074739+0000","last_clean":"2024-04-28T12:49:36.074739+0000","last_became_active":"2024-04-28T12:48:43.508596+0000","last_became_peered":"2024-04-28T12:48:43.508596+0000","last_unstale":"2024-04-28T12:49:36.074739+0000","last_undegraded":"2024-04-28T12:49:36.074739+0000","last_fullsized":"2024-04-28T12:49:36.074739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:41.905528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628233+0000","last_change":"2024-04-28T12:48:44.489695+0000","last_active":"2024-04-28T12:49:35.628233+0000","last_peered":"2024-04-28T12:49:35.628233+0000","last_clean":"2024-04-28T12:49:35.628233+0000","last_became_active":"2024-04-28T12:48:44.489290+0000","last_became_peered":"2024-04-28T12:48:44.489290+0000","last_unstale":"2024-04-28T12:49:35.628233+0000","last_undegraded":"2024-04-28T12:49:35.628233+0000","last_fullsized":"2024-04-28T12:49:35.628233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:34:01.873775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":92,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:57.558319+0000","last_change":"2024-04-28T12:48:43.511751+0000","last_active":"2024-04-28T12:49:57.558319+0000","last_peered":"2024-04-28T12:49:57.558319+0000","last_clean":"2024-04-28T12:49:57.558319+0000","last_became_active":"2024-04-28T12:48:43.511428+0000","last_became_peered":"2024-04-28T12:48:43.511428+0000","last_unstale":"2024-04-28T12:49:57.558319+0000","last_undegraded":"2024-04-28T12:49:57.558319+0000","last_fullsized":"2024-04-28T12:49:57.558319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:12.987697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.639163+0000","last_change":"2024-04-28T12:48:44.472310+0000","last_active":"2024-04-28T12:49:35.639163+0000","last_peered":"2024-04-28T12:49:35.639163+0000","last_clean":"2024-04-28T12:49:35.639163+0000","last_became_active":"2024-04-28T12:48:44.471978+0000","last_became_peered":"2024-04-28T12:48:44.471978+0000","last_unstale":"2024-04-28T12:49:35.639163+0000","last_undegraded":"2024-04-28T12:49:35.639163+0000","last_fullsized":"2024-04-28T12:49:35.639163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.068156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631642+0000","last_change":"2024-04-28T12:48:43.514475+0000","last_active":"2024-04-28T12:49:35.631642+0000","last_peered":"2024-04-28T12:49:35.631642+0000","last_clean":"2024-04-28T12:49:35.631642+0000","last_became_active":"2024-04-28T12:48:43.514067+0000","last_became_peered":"2024-04-28T12:48:43.514067+0000","last_unstale":"2024-04-28T12:49:35.631642+0000","last_undegraded":"2024-04-28T12:49:35.631642+0000","last_fullsized":"2024-04-28T12:49:35.631642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:52.125456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631188+0000","last_change":"2024-04-28T12:48:44.479210+0000","last_active":"2024-04-28T12:49:35.631188+0000","last_peered":"2024-04-28T12:49:35.631188+0000","last_clean":"2024-04-28T12:49:35.631188+0000","last_became_active":"2024-04-28T12:48:44.478816+0000","last_became_peered":"2024-04-28T12:48:44.478816+0000","last_unstale":"2024-04-28T12:49:35.631188+0000","last_undegraded":"2024-04-28T12:49:35.631188+0000","last_fullsized":"2024-04-28T12:49:35.631188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:03:06.994063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075237+0000","last_change":"2024-04-28T12:48:43.520207+0000","last_active":"2024-04-28T12:49:36.075237+0000","last_peered":"2024-04-28T12:49:36.075237+0000","last_clean":"2024-04-28T12:49:36.075237+0000","last_became_active":"2024-04-28T12:48:43.519823+0000","last_became_peered":"2024-04-28T12:48:43.519823+0000","last_unstale":"2024-04-28T12:49:36.075237+0000","last_undegraded":"2024-04-28T12:49:36.075237+0000","last_fullsized":"2024-04-28T12:49:36.075237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:02.238166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638709+0000","last_change":"2024-04-28T12:48:44.479632+0000","last_active":"2024-04-28T12:49:35.638709+0000","last_peered":"2024-04-28T12:49:35.638709+0000","last_clean":"2024-04-28T12:49:35.638709+0000","last_became_active":"2024-04-28T12:48:44.479197+0000","last_became_peered":"2024-04-28T12:48:44.479197+0000","last_unstale":"2024-04-28T12:49:35.638709+0000","last_undegraded":"2024-04-28T12:49:35.638709+0000","last_fullsized":"2024-04-28T12:49:35.638709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:49:01.557312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'25","reported_seq":208,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:59.116196+0000","last_change":"2024-04-28T12:49:36.083466+0000","last_active":"2024-04-28T12:49:59.116196+0000","last_peered":"2024-04-28T12:49:59.116196+0000","last_clean":"2024-04-28T12:49:59.116196+0000","last_became_active":"2024-04-28T12:49:36.076359+0000","last_became_peered":"2024-04-28T12:49:36.076359+0000","last_unstale":"2024-04-28T12:49:59.116196+0000","last_undegraded":"2024-04-28T12:49:59.116196+0000","last_fullsized":"2024-04-28T12:49:59.116196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:08.982295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50398,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":117,"num_write":26,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":50398,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635960+0000","last_change":"2024-04-28T12:48:44.484996+0000","last_active":"2024-04-28T12:49:35.635960+0000","last_peered":"2024-04-28T12:49:35.635960+0000","last_clean":"2024-04-28T12:49:35.635960+0000","last_became_active":"2024-04-28T12:48:44.484591+0000","last_became_peered":"2024-04-28T12:48:44.484591+0000","last_unstale":"2024-04-28T12:49:35.635960+0000","last_undegraded":"2024-04-28T12:49:35.635960+0000","last_fullsized":"2024-04-28T12:49:35.635960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:38:42.768757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633319+0000","last_change":"2024-04-28T12:48:43.510365+0000","last_active":"2024-04-28T12:49:35.633319+0000","last_peered":"2024-04-28T12:49:35.633319+0000","last_clean":"2024-04-28T12:49:35.633319+0000","last_became_active":"2024-04-28T12:48:43.510028+0000","last_became_peered":"2024-04-28T12:48:43.510028+0000","last_unstale":"2024-04-28T12:49:35.633319+0000","last_undegraded":"2024-04-28T12:49:35.633319+0000","last_fullsized":"2024-04-28T12:49:35.633319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:09:20.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074327+0000","last_change":"2024-04-28T12:48:44.489704+0000","last_active":"2024-04-28T12:49:36.074327+0000","last_peered":"2024-04-28T12:49:36.074327+0000","last_clean":"2024-04-28T12:49:36.074327+0000","last_became_active":"2024-04-28T12:48:44.489331+0000","last_became_peered":"2024-04-28T12:48:44.489331+0000","last_unstale":"2024-04-28T12:49:36.074327+0000","last_undegraded":"2024-04-28T12:49:36.074327+0000","last_fullsized":"2024-04-28T12:49:36.074327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:33.755818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074743+0000","last_change":"2024-04-28T12:48:43.519213+0000","last_active":"2024-04-28T12:49:36.074743+0000","last_peered":"2024-04-28T12:49:36.074743+0000","last_clean":"2024-04-28T12:49:36.074743+0000","last_became_active":"2024-04-28T12:48:43.518385+0000","last_became_peered":"2024-04-28T12:48:43.518385+0000","last_unstale":"2024-04-28T12:49:36.074743+0000","last_undegraded":"2024-04-28T12:49:36.074743+0000","last_fullsized":"2024-04-28T12:49:36.074743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:31:37.130504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631449+0000","last_change":"2024-04-28T12:48:44.480838+0000","last_active":"2024-04-28T12:49:35.631449+0000","last_peered":"2024-04-28T12:49:35.631449+0000","last_clean":"2024-04-28T12:49:35.631449+0000","last_became_active":"2024-04-28T12:48:44.480522+0000","last_became_peered":"2024-04-28T12:48:44.480522+0000","last_unstale":"2024-04-28T12:49:35.631449+0000","last_undegraded":"2024-04-28T12:49:35.631449+0000","last_fullsized":"2024-04-28T12:49:35.631449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:24.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:53.552334+0000","last_change":"2024-04-28T12:48:43.505498+0000","last_active":"2024-04-28T12:49:53.552334+0000","last_peered":"2024-04-28T12:49:53.552334+0000","last_clean":"2024-04-28T12:49:53.552334+0000","last_became_active":"2024-04-28T12:48:43.505041+0000","last_became_peered":"2024-04-28T12:48:43.505041+0000","last_unstale":"2024-04-28T12:49:53.552334+0000","last_undegraded":"2024-04-28T12:49:53.552334+0000","last_fullsized":"2024-04-28T12:49:53.552334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:36.624508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631059+0000","last_change":"2024-04-28T12:48:44.482698+0000","last_active":"2024-04-28T12:49:35.631059+0000","last_peered":"2024-04-28T12:49:35.631059+0000","last_clean":"2024-04-28T12:49:35.631059+0000","last_became_active":"2024-04-28T12:48:44.482365+0000","last_became_peered":"2024-04-28T12:48:44.482365+0000","last_unstale":"2024-04-28T12:49:35.631059+0000","last_undegraded":"2024-04-28T12:49:35.631059+0000","last_fullsized":"2024-04-28T12:49:35.631059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:45:33.648105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634552+0000","last_change":"2024-04-28T12:48:43.515942+0000","last_active":"2024-04-28T12:49:35.634552+0000","last_peered":"2024-04-28T12:49:35.634552+0000","last_clean":"2024-04-28T12:49:35.634552+0000","last_became_active":"2024-04-28T12:48:43.515497+0000","last_became_peered":"2024-04-28T12:48:43.515497+0000","last_unstale":"2024-04-28T12:49:35.634552+0000","last_undegraded":"2024-04-28T12:49:35.634552+0000","last_fullsized":"2024-04-28T12:49:35.634552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:39.489486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074619+0000","last_change":"2024-04-28T12:48:44.487831+0000","last_active":"2024-04-28T12:49:36.074619+0000","last_peered":"2024-04-28T12:49:36.074619+0000","last_clean":"2024-04-28T12:49:36.074619+0000","last_became_active":"2024-04-28T12:48:44.487341+0000","last_became_peered":"2024-04-28T12:48:44.487341+0000","last_unstale":"2024-04-28T12:49:36.074619+0000","last_undegraded":"2024-04-28T12:49:36.074619+0000","last_fullsized":"2024-04-28T12:49:36.074619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:42.810514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635784+0000","last_change":"2024-04-28T12:48:43.513667+0000","last_active":"2024-04-28T12:49:35.635784+0000","last_peered":"2024-04-28T12:49:35.635784+0000","last_clean":"2024-04-28T12:49:35.635784+0000","last_became_active":"2024-04-28T12:48:43.513192+0000","last_became_peered":"2024-04-28T12:48:43.513192+0000","last_unstale":"2024-04-28T12:49:35.635784+0000","last_undegraded":"2024-04-28T12:49:35.635784+0000","last_fullsized":"2024-04-28T12:49:35.635784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.470320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630764+0000","last_change":"2024-04-28T12:48:44.478203+0000","last_active":"2024-04-28T12:49:35.630764+0000","last_peered":"2024-04-28T12:49:35.630764+0000","last_clean":"2024-04-28T12:49:35.630764+0000","last_became_active":"2024-04-28T12:48:44.476238+0000","last_became_peered":"2024-04-28T12:48:44.476238+0000","last_unstale":"2024-04-28T12:49:35.630764+0000","last_undegraded":"2024-04-28T12:49:35.630764+0000","last_fullsized":"2024-04-28T12:49:35.630764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:46:49.810015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628368+0000","last_change":"2024-04-28T12:48:43.516036+0000","last_active":"2024-04-28T12:49:35.628368+0000","last_peered":"2024-04-28T12:49:35.628368+0000","last_clean":"2024-04-28T12:49:35.628368+0000","last_became_active":"2024-04-28T12:48:43.515494+0000","last_became_peered":"2024-04-28T12:48:43.515494+0000","last_unstale":"2024-04-28T12:49:35.628368+0000","last_undegraded":"2024-04-28T12:49:35.628368+0000","last_fullsized":"2024-04-28T12:49:35.628368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:34:56.568179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634892+0000","last_change":"2024-04-28T12:48:44.476836+0000","last_active":"2024-04-28T12:49:35.634892+0000","last_peered":"2024-04-28T12:49:35.634892+0000","last_clean":"2024-04-28T12:49:35.634892+0000","last_became_active":"2024-04-28T12:48:44.476214+0000","last_became_peered":"2024-04-28T12:48:44.476214+0000","last_unstale":"2024-04-28T12:49:35.634892+0000","last_undegraded":"2024-04-28T12:49:35.634892+0000","last_fullsized":"2024-04-28T12:49:35.634892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:24:26.707985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631213+0000","last_change":"2024-04-28T12:48:43.520918+0000","last_active":"2024-04-28T12:49:35.631213+0000","last_peered":"2024-04-28T12:49:35.631213+0000","last_clean":"2024-04-28T12:49:35.631213+0000","last_became_active":"2024-04-28T12:48:43.520564+0000","last_became_peered":"2024-04-28T12:48:43.520564+0000","last_unstale":"2024-04-28T12:49:35.631213+0000","last_undegraded":"2024-04-28T12:49:35.631213+0000","last_fullsized":"2024-04-28T12:49:35.631213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:32:51.206102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635059+0000","last_change":"2024-04-28T12:48:44.484570+0000","last_active":"2024-04-28T12:49:35.635059+0000","last_peered":"2024-04-28T12:49:35.635059+0000","last_clean":"2024-04-28T12:49:35.635059+0000","last_became_active":"2024-04-28T12:48:44.484133+0000","last_became_peered":"2024-04-28T12:48:44.484133+0000","last_unstale":"2024-04-28T12:49:35.635059+0000","last_undegraded":"2024-04-28T12:49:35.635059+0000","last_fullsized":"2024-04-28T12:49:35.635059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:59:14.274873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074978+0000","last_change":"2024-04-28T12:48:44.478013+0000","last_active":"2024-04-28T12:49:36.074978+0000","last_peered":"2024-04-28T12:49:36.074978+0000","last_clean":"2024-04-28T12:49:36.074978+0000","last_became_active":"2024-04-28T12:48:44.477559+0000","last_became_peered":"2024-04-28T12:48:44.477559+0000","last_unstale":"2024-04-28T12:49:36.074978+0000","last_undegraded":"2024-04-28T12:49:36.074978+0000","last_fullsized":"2024-04-28T12:49:36.074978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:57:49.931865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:54.553143+0000","last_change":"2024-04-28T12:48:43.515873+0000","last_active":"2024-04-28T12:49:54.553143+0000","last_peered":"2024-04-28T12:49:54.553143+0000","last_clean":"2024-04-28T12:49:54.553143+0000","last_became_active":"2024-04-28T12:48:43.515442+0000","last_became_peered":"2024-04-28T12:48:43.515442+0000","last_unstale":"2024-04-28T12:49:54.553143+0000","last_undegraded":"2024-04-28T12:49:54.553143+0000","last_fullsized":"2024-04-28T12:49:54.553143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:29:57.973433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811001+0000","last_change":"2024-04-28T12:48:44.479868+0000","last_active":"2024-04-28T12:49:35.811001+0000","last_peered":"2024-04-28T12:49:35.811001+0000","last_clean":"2024-04-28T12:49:35.811001+0000","last_became_active":"2024-04-28T12:48:44.479519+0000","last_became_peered":"2024-04-28T12:48:44.479519+0000","last_unstale":"2024-04-28T12:49:35.811001+0000","last_undegraded":"2024-04-28T12:49:35.811001+0000","last_fullsized":"2024-04-28T12:49:35.811001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:09:20.946380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074568+0000","last_change":"2024-04-28T12:48:43.507662+0000","last_active":"2024-04-28T12:49:36.074568+0000","last_peered":"2024-04-28T12:49:36.074568+0000","last_clean":"2024-04-28T12:49:36.074568+0000","last_became_active":"2024-04-28T12:48:43.507331+0000","last_became_peered":"2024-04-28T12:48:43.507331+0000","last_unstale":"2024-04-28T12:49:36.074568+0000","last_undegraded":"2024-04-28T12:49:36.074568+0000","last_fullsized":"2024-04-28T12:49:36.074568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:10.073569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":342,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631169+0000","last_change":"2024-04-28T12:47:27.444407+0000","last_active":"2024-04-28T12:49:35.631169+0000","last_peered":"2024-04-28T12:49:35.631169+0000","last_clean":"2024-04-28T12:49:35.631169+0000","last_became_active":"2024-04-28T12:47:27.443992+0000","last_became_peered":"2024-04-28T12:47:27.443992+0000","last_unstale":"2024-04-28T12:49:35.631169+0000","last_undegraded":"2024-04-28T12:49:35.631169+0000","last_fullsized":"2024-04-28T12:49:35.631169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_clean_scrub_stamp":"2024-04-28T12:44:00.636011+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:17:23.614524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074164+0000","last_change":"2024-04-28T12:48:44.479395+0000","last_active":"2024-04-28T12:49:36.074164+0000","last_peered":"2024-04-28T12:49:36.074164+0000","last_clean":"2024-04-28T12:49:36.074164+0000","last_became_active":"2024-04-28T12:48:44.478935+0000","last_became_peered":"2024-04-28T12:48:44.478935+0000","last_unstale":"2024-04-28T12:49:36.074164+0000","last_undegraded":"2024-04-28T12:49:36.074164+0000","last_fullsized":"2024-04-28T12:49:36.074164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:19:28.212744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631257+0000","last_change":"2024-04-28T12:48:43.512026+0000","last_active":"2024-04-28T12:49:35.631257+0000","last_peered":"2024-04-28T12:49:35.631257+0000","last_clean":"2024-04-28T12:49:35.631257+0000","last_became_active":"2024-04-28T12:48:43.511650+0000","last_became_peered":"2024-04-28T12:48:43.511650+0000","last_unstale":"2024-04-28T12:49:35.631257+0000","last_undegraded":"2024-04-28T12:49:35.631257+0000","last_fullsized":"2024-04-28T12:49:35.631257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:38:04.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.641820+0000","last_change":"2024-04-28T12:49:35.641378+0000","last_active":"2024-04-28T12:49:35.641820+0000","last_peered":"2024-04-28T12:49:35.641820+0000","last_clean":"2024-04-28T12:49:35.641820+0000","last_became_active":"2024-04-28T12:49:35.633729+0000","last_became_peered":"2024-04-28T12:49:35.633729+0000","last_unstale":"2024-04-28T12:49:35.641820+0000","last_undegraded":"2024-04-28T12:49:35.641820+0000","last_fullsized":"2024-04-28T12:49:35.641820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:16:22.142685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074413+0000","last_change":"2024-04-28T12:48:44.470458+0000","last_active":"2024-04-28T12:49:36.074413+0000","last_peered":"2024-04-28T12:49:36.074413+0000","last_clean":"2024-04-28T12:49:36.074413+0000","last_became_active":"2024-04-28T12:48:44.470105+0000","last_became_peered":"2024-04-28T12:48:44.470105+0000","last_unstale":"2024-04-28T12:49:36.074413+0000","last_undegraded":"2024-04-28T12:49:36.074413+0000","last_fullsized":"2024-04-28T12:49:36.074413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:49:25.970990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628468+0000","last_change":"2024-04-28T12:48:43.510065+0000","last_active":"2024-04-28T12:49:35.628468+0000","last_peered":"2024-04-28T12:49:35.628468+0000","last_clean":"2024-04-28T12:49:35.628468+0000","last_became_active":"2024-04-28T12:48:43.509718+0000","last_became_peered":"2024-04-28T12:48:43.509718+0000","last_unstale":"2024-04-28T12:49:35.628468+0000","last_undegraded":"2024-04-28T12:49:35.628468+0000","last_fullsized":"2024-04-28T12:49:35.628468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:50:46.776100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811200+0000","last_change":"2024-04-28T12:48:44.487022+0000","last_active":"2024-04-28T12:49:35.811200+0000","last_peered":"2024-04-28T12:49:35.811200+0000","last_clean":"2024-04-28T12:49:35.811200+0000","last_became_active":"2024-04-28T12:48:44.486675+0000","last_became_peered":"2024-04-28T12:48:44.486675+0000","last_unstale":"2024-04-28T12:49:35.811200+0000","last_undegraded":"2024-04-28T12:49:35.811200+0000","last_fullsized":"2024-04-28T12:49:35.811200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:36.185845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075014+0000","last_change":"2024-04-28T12:48:43.505464+0000","last_active":"2024-04-28T12:49:36.075014+0000","last_peered":"2024-04-28T12:49:36.075014+0000","last_clean":"2024-04-28T12:49:36.075014+0000","last_became_active":"2024-04-28T12:48:43.505085+0000","last_became_peered":"2024-04-28T12:48:43.505085+0000","last_unstale":"2024-04-28T12:49:36.075014+0000","last_undegraded":"2024-04-28T12:49:36.075014+0000","last_fullsized":"2024-04-28T12:49:36.075014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:37:47.875243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.812460+0000","last_change":"2024-04-28T12:49:35.812460+0000","last_active":"2024-04-28T12:49:35.812460+0000","last_peered":"2024-04-28T12:49:35.812460+0000","last_clean":"2024-04-28T12:49:35.812460+0000","last_became_active":"2024-04-28T12:49:35.812047+0000","last_became_peered":"2024-04-28T12:49:35.812047+0000","last_unstale":"2024-04-28T12:49:35.812460+0000","last_undegraded":"2024-04-28T12:49:35.812460+0000","last_fullsized":"2024-04-28T12:49:35.812460+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:22:47.979802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638144+0000","last_change":"2024-04-28T12:48:43.508992+0000","last_active":"2024-04-28T12:49:35.638144+0000","last_peered":"2024-04-28T12:49:35.638144+0000","last_clean":"2024-04-28T12:49:35.638144+0000","last_became_active":"2024-04-28T12:48:43.508492+0000","last_became_peered":"2024-04-28T12:48:43.508492+0000","last_unstale":"2024-04-28T12:49:35.638144+0000","last_undegraded":"2024-04-28T12:49:35.638144+0000","last_fullsized":"2024-04-28T12:49:35.638144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:56:29.138232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634927+0000","last_change":"2024-04-28T12:48:44.476462+0000","last_active":"2024-04-28T12:49:35.634927+0000","last_peered":"2024-04-28T12:49:35.634927+0000","last_clean":"2024-04-28T12:49:35.634927+0000","last_became_active":"2024-04-28T12:48:44.475660+0000","last_became_peered":"2024-04-28T12:48:44.475660+0000","last_unstale":"2024-04-28T12:49:35.634927+0000","last_undegraded":"2024-04-28T12:49:35.634927+0000","last_fullsized":"2024-04-28T12:49:35.634927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:49:48.656837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810825+0000","last_change":"2024-04-28T12:48:43.508946+0000","last_active":"2024-04-28T12:49:35.810825+0000","last_peered":"2024-04-28T12:49:35.810825+0000","last_clean":"2024-04-28T12:49:35.810825+0000","last_became_active":"2024-04-28T12:48:43.508553+0000","last_became_peered":"2024-04-28T12:48:43.508553+0000","last_unstale":"2024-04-28T12:49:35.810825+0000","last_undegraded":"2024-04-28T12:49:35.810825+0000","last_fullsized":"2024-04-28T12:49:35.810825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:52:56.461509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630544+0000","last_change":"2024-04-28T12:48:44.473967+0000","last_active":"2024-04-28T12:49:35.630544+0000","last_peered":"2024-04-28T12:49:35.630544+0000","last_clean":"2024-04-28T12:49:35.630544+0000","last_became_active":"2024-04-28T12:48:44.473614+0000","last_became_peered":"2024-04-28T12:48:44.473614+0000","last_unstale":"2024-04-28T12:49:35.630544+0000","last_undegraded":"2024-04-28T12:49:35.630544+0000","last_fullsized":"2024-04-28T12:49:35.630544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:58:20.994924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633162+0000","last_change":"2024-04-28T12:48:43.511287+0000","last_active":"2024-04-28T12:49:35.633162+0000","last_peered":"2024-04-28T12:49:35.633162+0000","last_clean":"2024-04-28T12:49:35.633162+0000","last_became_active":"2024-04-28T12:48:43.510955+0000","last_became_peered":"2024-04-28T12:48:43.510955+0000","last_unstale":"2024-04-28T12:49:35.633162+0000","last_undegraded":"2024-04-28T12:49:35.633162+0000","last_fullsized":"2024-04-28T12:49:35.633162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:40.952405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811070+0000","last_change":"2024-04-28T12:48:44.480514+0000","last_active":"2024-04-28T12:49:35.811070+0000","last_peered":"2024-04-28T12:49:35.811070+0000","last_clean":"2024-04-28T12:49:35.811070+0000","last_became_active":"2024-04-28T12:48:44.480160+0000","last_became_peered":"2024-04-28T12:48:44.480160+0000","last_unstale":"2024-04-28T12:49:35.811070+0000","last_undegraded":"2024-04-28T12:49:35.811070+0000","last_fullsized":"2024-04-28T12:49:35.811070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:07:02.221492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.643739+0000","last_change":"2024-04-28T12:49:35.643739+0000","last_active":"2024-04-28T12:49:35.643739+0000","last_peered":"2024-04-28T12:49:35.643739+0000","last_clean":"2024-04-28T12:49:35.643739+0000","last_became_active":"2024-04-28T12:49:35.639497+0000","last_became_peered":"2024-04-28T12:49:35.639497+0000","last_unstale":"2024-04-28T12:49:35.643739+0000","last_undegraded":"2024-04-28T12:49:35.643739+0000","last_fullsized":"2024-04-28T12:49:35.643739+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:59:52.886324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631343+0000","last_change":"2024-04-28T12:48:44.474095+0000","last_active":"2024-04-28T12:49:35.631343+0000","last_peered":"2024-04-28T12:49:35.631343+0000","last_clean":"2024-04-28T12:49:35.631343+0000","last_became_active":"2024-04-28T12:48:44.473377+0000","last_became_peered":"2024-04-28T12:48:44.473377+0000","last_unstale":"2024-04-28T12:49:35.631343+0000","last_undegraded":"2024-04-28T12:49:35.631343+0000","last_fullsized":"2024-04-28T12:49:35.631343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.355527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634193+0000","last_change":"2024-04-28T12:48:43.515654+0000","last_active":"2024-04-28T12:49:35.634193+0000","last_peered":"2024-04-28T12:49:35.634193+0000","last_clean":"2024-04-28T12:49:35.634193+0000","last_became_active":"2024-04-28T12:48:43.515056+0000","last_became_peered":"2024-04-28T12:48:43.515056+0000","last_unstale":"2024-04-28T12:49:35.634193+0000","last_undegraded":"2024-04-28T12:49:35.634193+0000","last_fullsized":"2024-04-28T12:49:35.634193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:47:58.297248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631641+0000","last_change":"2024-04-28T12:48:44.474106+0000","last_active":"2024-04-28T12:49:35.631641+0000","last_peered":"2024-04-28T12:49:35.631641+0000","last_clean":"2024-04-28T12:49:35.631641+0000","last_became_active":"2024-04-28T12:48:44.473625+0000","last_became_peered":"2024-04-28T12:48:44.473625+0000","last_unstale":"2024-04-28T12:49:35.631641+0000","last_undegraded":"2024-04-28T12:49:35.631641+0000","last_fullsized":"2024-04-28T12:49:35.631641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:28:44.434155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075075+0000","last_change":"2024-04-28T12:48:43.519658+0000","last_active":"2024-04-28T12:49:36.075075+0000","last_peered":"2024-04-28T12:49:36.075075+0000","last_clean":"2024-04-28T12:49:36.075075+0000","last_became_active":"2024-04-28T12:48:43.519291+0000","last_became_peered":"2024-04-28T12:48:43.519291+0000","last_unstale":"2024-04-28T12:49:36.075075+0000","last_undegraded":"2024-04-28T12:49:36.075075+0000","last_fullsized":"2024-04-28T12:49:36.075075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:41.356834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631413+0000","last_change":"2024-04-28T12:48:44.483790+0000","last_active":"2024-04-28T12:49:35.631413+0000","last_peered":"2024-04-28T12:49:35.631413+0000","last_clean":"2024-04-28T12:49:35.631413+0000","last_became_active":"2024-04-28T12:48:44.483442+0000","last_became_peered":"2024-04-28T12:48:44.483442+0000","last_unstale":"2024-04-28T12:49:35.631413+0000","last_undegraded":"2024-04-28T12:49:35.631413+0000","last_fullsized":"2024-04-28T12:49:35.631413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:38:59.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074428+0000","last_change":"2024-04-28T12:48:43.505650+0000","last_active":"2024-04-28T12:49:36.074428+0000","last_peered":"2024-04-28T12:49:36.074428+0000","last_clean":"2024-04-28T12:49:36.074428+0000","last_became_active":"2024-04-28T12:48:43.505305+0000","last_became_peered":"2024-04-28T12:48:43.505305+0000","last_unstale":"2024-04-28T12:49:36.074428+0000","last_undegraded":"2024-04-28T12:49:36.074428+0000","last_fullsized":"2024-04-28T12:49:36.074428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:56:34.654105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631739+0000","last_change":"2024-04-28T12:48:44.478335+0000","last_active":"2024-04-28T12:49:35.631739+0000","last_peered":"2024-04-28T12:49:35.631739+0000","last_clean":"2024-04-28T12:49:35.631739+0000","last_became_active":"2024-04-28T12:48:44.477958+0000","last_became_peered":"2024-04-28T12:48:44.477958+0000","last_unstale":"2024-04-28T12:49:35.631739+0000","last_undegraded":"2024-04-28T12:49:35.631739+0000","last_fullsized":"2024-04-28T12:49:35.631739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:27.168011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.637326+0000","last_change":"2024-04-28T12:49:35.637326+0000","last_active":"2024-04-28T12:49:35.637326+0000","last_peered":"2024-04-28T12:49:35.637326+0000","last_clean":"2024-04-28T12:49:35.637326+0000","last_became_active":"2024-04-28T12:49:35.636959+0000","last_became_peered":"2024-04-28T12:49:35.636959+0000","last_unstale":"2024-04-28T12:49:35.637326+0000","last_undegraded":"2024-04-28T12:49:35.637326+0000","last_fullsized":"2024-04-28T12:49:35.637326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:40:22.428732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633418+0000","last_change":"2024-04-28T12:48:44.478553+0000","last_active":"2024-04-28T12:49:35.633418+0000","last_peered":"2024-04-28T12:49:35.633418+0000","last_clean":"2024-04-28T12:49:35.633418+0000","last_became_active":"2024-04-28T12:48:44.477539+0000","last_became_peered":"2024-04-28T12:48:44.477539+0000","last_unstale":"2024-04-28T12:49:35.633418+0000","last_undegraded":"2024-04-28T12:49:35.633418+0000","last_fullsized":"2024-04-28T12:49:35.633418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:03:44.149337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635856+0000","last_change":"2024-04-28T12:49:35.635856+0000","last_active":"2024-04-28T12:49:35.635856+0000","last_peered":"2024-04-28T12:49:35.635856+0000","last_clean":"2024-04-28T12:49:35.635856+0000","last_became_active":"2024-04-28T12:49:35.635343+0000","last_became_peered":"2024-04-28T12:49:35.635343+0000","last_unstale":"2024-04-28T12:49:35.635856+0000","last_undegraded":"2024-04-28T12:49:35.635856+0000","last_fullsized":"2024-04-28T12:49:35.635856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:32:09.319536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630843+0000","last_change":"2024-04-28T12:48:44.479729+0000","last_active":"2024-04-28T12:49:35.630843+0000","last_peered":"2024-04-28T12:49:35.630843+0000","last_clean":"2024-04-28T12:49:35.630843+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:49:35.630843+0000","last_undegraded":"2024-04-28T12:49:35.630843+0000","last_fullsized":"2024-04-28T12:49:35.630843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:33:32.456549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074522+0000","last_change":"2024-04-28T12:48:44.486785+0000","last_active":"2024-04-28T12:49:36.074522+0000","last_peered":"2024-04-28T12:49:36.074522+0000","last_clean":"2024-04-28T12:49:36.074522+0000","last_became_active":"2024-04-28T12:48:44.485634+0000","last_became_peered":"2024-04-28T12:48:44.485634+0000","last_unstale":"2024-04-28T12:49:36.074522+0000","last_undegraded":"2024-04-28T12:49:36.074522+0000","last_fullsized":"2024-04-28T12:49:36.074522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:58.148545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631237+0000","last_change":"2024-04-28T12:48:44.479690+0000","last_active":"2024-04-28T12:49:35.631237+0000","last_peered":"2024-04-28T12:49:35.631237+0000","last_clean":"2024-04-28T12:49:35.631237+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:49:35.631237+0000","last_undegraded":"2024-04-28T12:49:35.631237+0000","last_fullsized":"2024-04-28T12:49:35.631237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:40:47.274081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633795+0000","last_change":"2024-04-28T12:48:44.472305+0000","last_active":"2024-04-28T12:49:35.633795+0000","last_peered":"2024-04-28T12:49:35.633795+0000","last_clean":"2024-04-28T12:49:35.633795+0000","last_became_active":"2024-04-28T12:48:44.471227+0000","last_became_peered":"2024-04-28T12:48:44.471227+0000","last_unstale":"2024-04-28T12:49:35.633795+0000","last_undegraded":"2024-04-28T12:49:35.633795+0000","last_fullsized":"2024-04-28T12:49:35.633795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:27:14.602075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811084+0000","last_change":"2024-04-28T12:48:44.490866+0000","last_active":"2024-04-28T12:49:35.811084+0000","last_peered":"2024-04-28T12:49:35.811084+0000","last_clean":"2024-04-28T12:49:35.811084+0000","last_became_active":"2024-04-28T12:48:44.490515+0000","last_became_peered":"2024-04-28T12:48:44.490515+0000","last_unstale":"2024-04-28T12:49:35.811084+0000","last_undegraded":"2024-04-28T12:49:35.811084+0000","last_fullsized":"2024-04-28T12:49:35.811084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:19:22.404853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074784+0000","last_change":"2024-04-28T12:48:44.480611+0000","last_active":"2024-04-28T12:49:36.074784+0000","last_peered":"2024-04-28T12:49:36.074784+0000","last_clean":"2024-04-28T12:49:36.074784+0000","last_became_active":"2024-04-28T12:48:44.480294+0000","last_became_peered":"2024-04-28T12:48:44.480294+0000","last_unstale":"2024-04-28T12:49:36.074784+0000","last_undegraded":"2024-04-28T12:49:36.074784+0000","last_fullsized":"2024-04-28T12:49:36.074784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:55.450825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810781+0000","last_change":"2024-04-28T12:48:44.483003+0000","last_active":"2024-04-28T12:49:35.810781+0000","last_peered":"2024-04-28T12:49:35.810781+0000","last_clean":"2024-04-28T12:49:35.810781+0000","last_became_active":"2024-04-28T12:48:44.482630+0000","last_became_peered":"2024-04-28T12:48:44.482630+0000","last_unstale":"2024-04-28T12:49:35.810781+0000","last_undegraded":"2024-04-28T12:49:35.810781+0000","last_fullsized":"2024-04-28T12:49:35.810781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:05:58.358655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638350+0000","last_change":"2024-04-28T12:48:44.489780+0000","last_active":"2024-04-28T12:49:35.638350+0000","last_peered":"2024-04-28T12:49:35.638350+0000","last_clean":"2024-04-28T12:49:35.638350+0000","last_became_active":"2024-04-28T12:48:44.489276+0000","last_became_peered":"2024-04-28T12:48:44.489276+0000","last_unstale":"2024-04-28T12:49:35.638350+0000","last_undegraded":"2024-04-28T12:49:35.638350+0000","last_fullsized":"2024-04-28T12:49:35.638350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:49.264624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074471+0000","last_change":"2024-04-28T12:48:44.472196+0000","last_active":"2024-04-28T12:49:36.074471+0000","last_peered":"2024-04-28T12:49:36.074471+0000","last_clean":"2024-04-28T12:49:36.074471+0000","last_became_active":"2024-04-28T12:48:44.471857+0000","last_became_peered":"2024-04-28T12:48:44.471857+0000","last_unstale":"2024-04-28T12:49:36.074471+0000","last_undegraded":"2024-04-28T12:49:36.074471+0000","last_fullsized":"2024-04-28T12:49:36.074471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:43.813225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630551+0000","last_change":"2024-04-28T12:48:44.478788+0000","last_active":"2024-04-28T12:49:35.630551+0000","last_peered":"2024-04-28T12:49:35.630551+0000","last_clean":"2024-04-28T12:49:35.630551+0000","last_became_active":"2024-04-28T12:48:44.478449+0000","last_became_peered":"2024-04-28T12:48:44.478449+0000","last_unstale":"2024-04-28T12:49:35.630551+0000","last_undegraded":"2024-04-28T12:49:35.630551+0000","last_fullsized":"2024-04-28T12:49:35.630551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:48:11.638628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283884,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54008,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":428,"num_read_kb":271,"num_write":152,"num_write_kb":95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":50420,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":162024,"data_compressed":11972,"data_compressed_allocated":20480,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":459357,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1101214,"data_compressed":43734,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":966656,"data_stored":459571,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1598,"internal_metadata":36174274},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":459469,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299442,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1051030,"data_compressed":41890,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1587,"internal_metadata":31914445},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":458960,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":509151,"data_compressed":7635,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":460231,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460103,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":459271,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888129,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":509795,"data_compressed":14075,"data_compressed_allocated":20480,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":1050714,"data_compressed":41890,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":50398,"data_compressed":1844,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":51042,"data_compressed":8284,"data_compressed_allocated":12288,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":51976,"data_compressed":1844,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:50:01.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-28T12:50:01.760 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:01 smithi077 ceph-mon[32553]: pgmap v331: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:01 smithi077 ceph-mon[32553]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:50:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:50:01.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:01 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:01.974 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:01 smithi053 ceph-mon[29182]: pgmap v331: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:01.975 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:01.975 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:01.975 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:01 smithi053 ceph-mon[29182]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:01.975 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:50:01.975 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:50:01.975 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:01 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:01 smithi191 ceph-mon[33016]: pgmap v331: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:01 smithi191 ceph-mon[33016]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:50:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:50:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:01 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:50:03.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 -- 172.21.15.53:0/3872438521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103e30 msgr2=0x7f5748110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:03.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 --2- 172.21.15.53:0/3872438521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103e30 0x7f5748110290 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f574000b0a0 tx=0x7f574000b3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 -- 172.21.15.53:0/3872438521 shutdown_connections 2024-04-28T12:50:03.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 --2- 172.21.15.53:0/3872438521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103e30 0x7f5748110290 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 --2- 172.21.15.53:0/3872438521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57481094c0 0x7f57481098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 --2- 172.21.15.53:0/3872438521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5748103470 0x7f57481038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 -- 172.21.15.53:0/3872438521 >> 172.21.15.53:0/3872438521 conn(0x7f574807a580 msgr2=0x7f574807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 -- 172.21.15.53:0/3872438521 shutdown_connections 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 -- 172.21.15.53:0/3872438521 wait complete. 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.463+0000 7f574d759700 1 Processor -- start 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f574d759700 1 -- start start 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f574d759700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103470 0x7f57481a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f574d759700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5748103e30 0x7f57481a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f574d759700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57481094c0 0x7f574819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f574d759700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5748113630 con 0x7f5748103470 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f574d759700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57481134b0 con 0x7f5748103e30 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f574d759700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f57481137b0 con 0x7f57481094c0 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103470 0x7f57481a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:03.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103470 0x7f57481a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43704/0 (socket says 172.21.15.53:43704) 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 -- 172.21.15.53:0/2686454351 learned_addr learned my addr 172.21.15.53:0/2686454351 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f57467fc700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5748103e30 0x7f57481a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f57477fe700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57481094c0 0x7f574819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 -- 172.21.15.53:0/2686454351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57481094c0 msgr2=0x7f574819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57481094c0 0x7f574819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 -- 172.21.15.53:0/2686454351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5748103e30 msgr2=0x7f57481a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5748103e30 0x7f57481a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 -- 172.21.15.53:0/2686454351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574819c410 con 0x7f5748103470 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.464+0000 7f5746ffd700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103470 0x7f57481a1a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f573800c970 tx=0x7f573800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.465+0000 7f572ffff700 1 -- 172.21.15.53:0/2686454351 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738007aa0 con 0x7f5748103470 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.465+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574819c700 con 0x7f5748103470 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.465+0000 7f572ffff700 1 -- 172.21.15.53:0/2686454351 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5738004ce0 con 0x7f5748103470 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.465+0000 7f572ffff700 1 -- 172.21.15.53:0/2686454351 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738005510 con 0x7f5748103470 2024-04-28T12:50:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.465+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57481a89e0 con 0x7f5748103470 2024-04-28T12:50:03.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.466+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5748104b60 con 0x7f5748103470 2024-04-28T12:50:03.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.467+0000 7f572ffff700 1 -- 172.21.15.53:0/2686454351 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5738007c40 con 0x7f5748103470 2024-04-28T12:50:03.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.467+0000 7f572ffff700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f57300a3cd0 0x7f57300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:03.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.468+0000 7f57467fc700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f57300a3cd0 0x7f57300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:03.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.468+0000 7f572ffff700 1 -- 172.21.15.53:0/2686454351 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f57380d02e0 con 0x7f5748103470 2024-04-28T12:50:03.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.468+0000 7f57467fc700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f57300a3cd0 0x7f57300a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f574819d280 tx=0x7f573c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:03.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.472+0000 7f572ffff700 1 -- 172.21.15.53:0/2686454351 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f573809cbb0 con 0x7f5748103470 2024-04-28T12:50:03.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.677+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f574819cff0 con 0x7f57300a3cd0 2024-04-28T12:50:03.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.686+0000 7f572ffff700 1 -- 172.21.15.53:0/2686454351 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259021 (secure 0 0 0) 0x7f574819cff0 con 0x7f57300a3cd0 2024-04-28T12:50:03.686 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f57300a3cd0 msgr2=0x7f57300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f57300a3cd0 0x7f57300a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f574819d280 tx=0x7f573c0073d0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103470 msgr2=0x7f57481a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103470 0x7f57481a1a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f573800c970 tx=0x7f573800cc80 comp rx=0 tx=0).stop 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 shutdown_connections 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f57300a3cd0 0x7f57300a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5748103470 0x7f57481a1a60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57481094c0 0x7f574819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 --2- 172.21.15.53:0/2686454351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5748103e30 0x7f57481a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 >> 172.21.15.53:0/2686454351 conn(0x7f574807a580 msgr2=0x7f574810e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 shutdown_connections 2024-04-28T12:50:03.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:03.689+0000 7f574d759700 1 -- 172.21.15.53:0/2686454351 wait complete. 2024-04-28T12:50:03.689 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-04-28T12:50:03.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:03 smithi077 ceph-mon[32553]: pgmap v332: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:03 smithi191 ceph-mon[33016]: pgmap v332: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:04.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:03 smithi053 ceph-mon[29182]: pgmap v332: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:04.322 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-04-28T12:50:02.470428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644610,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":564,"num_read_kb":385,"num_write":335,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1231156,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377460,"kb_used_data":11596,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564364,"statfs":{"total":1151940427776,"available":1151553908736,"internally_reserved":0,"allocated":11874304,"data_stored":7438866,"data_compressed":189761,"data_compressed_allocated":253952,"data_compressed_original":2031616,"omap_allocated":19119,"internal_metadata":374453585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003491"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630828+0000","last_change":"2024-04-28T12:48:44.486823+0000","last_active":"2024-04-28T12:49:35.630828+0000","last_peered":"2024-04-28T12:49:35.630828+0000","last_clean":"2024-04-28T12:49:35.630828+0000","last_became_active":"2024-04-28T12:48:44.485159+0000","last_became_peered":"2024-04-28T12:48:44.485159+0000","last_unstale":"2024-04-28T12:49:35.630828+0000","last_undegraded":"2024-04-28T12:49:35.630828+0000","last_fullsized":"2024-04-28T12:49:35.630828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:12.126908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630923+0000","last_change":"2024-04-28T12:48:44.477801+0000","last_active":"2024-04-28T12:49:35.630923+0000","last_peered":"2024-04-28T12:49:35.630923+0000","last_clean":"2024-04-28T12:49:35.630923+0000","last_became_active":"2024-04-28T12:48:44.477448+0000","last_became_peered":"2024-04-28T12:48:44.477448+0000","last_unstale":"2024-04-28T12:49:35.630923+0000","last_undegraded":"2024-04-28T12:49:35.630923+0000","last_fullsized":"2024-04-28T12:49:35.630923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:30.350306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630895+0000","last_change":"2024-04-28T12:48:44.485638+0000","last_active":"2024-04-28T12:49:35.630895+0000","last_peered":"2024-04-28T12:49:35.630895+0000","last_clean":"2024-04-28T12:49:35.630895+0000","last_became_active":"2024-04-28T12:48:44.485247+0000","last_became_peered":"2024-04-28T12:48:44.485247+0000","last_unstale":"2024-04-28T12:49:35.630895+0000","last_undegraded":"2024-04-28T12:49:35.630895+0000","last_fullsized":"2024-04-28T12:49:35.630895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:49:35.209786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075121+0000","last_change":"2024-04-28T12:48:44.484329+0000","last_active":"2024-04-28T12:49:36.075121+0000","last_peered":"2024-04-28T12:49:36.075121+0000","last_clean":"2024-04-28T12:49:36.075121+0000","last_became_active":"2024-04-28T12:48:44.483980+0000","last_became_peered":"2024-04-28T12:48:44.483980+0000","last_unstale":"2024-04-28T12:49:36.075121+0000","last_undegraded":"2024-04-28T12:49:36.075121+0000","last_fullsized":"2024-04-28T12:49:36.075121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:14:47.570749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635150+0000","last_change":"2024-04-28T12:48:44.487609+0000","last_active":"2024-04-28T12:49:35.635150+0000","last_peered":"2024-04-28T12:49:35.635150+0000","last_clean":"2024-04-28T12:49:35.635150+0000","last_became_active":"2024-04-28T12:48:44.487199+0000","last_became_peered":"2024-04-28T12:48:44.487199+0000","last_unstale":"2024-04-28T12:49:35.635150+0000","last_undegraded":"2024-04-28T12:49:35.635150+0000","last_fullsized":"2024-04-28T12:49:35.635150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:30:03.675503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634772+0000","last_change":"2024-04-28T12:48:44.477889+0000","last_active":"2024-04-28T12:49:35.634772+0000","last_peered":"2024-04-28T12:49:35.634772+0000","last_clean":"2024-04-28T12:49:35.634772+0000","last_became_active":"2024-04-28T12:48:44.477472+0000","last_became_peered":"2024-04-28T12:48:44.477472+0000","last_unstale":"2024-04-28T12:49:35.634772+0000","last_undegraded":"2024-04-28T12:49:35.634772+0000","last_fullsized":"2024-04-28T12:49:35.634772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:35:03.684380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631190+0000","last_change":"2024-04-28T12:48:44.482889+0000","last_active":"2024-04-28T12:49:35.631190+0000","last_peered":"2024-04-28T12:49:35.631190+0000","last_clean":"2024-04-28T12:49:35.631190+0000","last_became_active":"2024-04-28T12:48:44.482569+0000","last_became_peered":"2024-04-28T12:48:44.482569+0000","last_unstale":"2024-04-28T12:49:35.631190+0000","last_undegraded":"2024-04-28T12:49:35.631190+0000","last_fullsized":"2024-04-28T12:49:35.631190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:07.858792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810695+0000","last_change":"2024-04-28T12:48:44.484118+0000","last_active":"2024-04-28T12:49:35.810695+0000","last_peered":"2024-04-28T12:49:35.810695+0000","last_clean":"2024-04-28T12:49:35.810695+0000","last_became_active":"2024-04-28T12:48:44.483577+0000","last_became_peered":"2024-04-28T12:48:44.483577+0000","last_unstale":"2024-04-28T12:49:35.810695+0000","last_undegraded":"2024-04-28T12:49:35.810695+0000","last_fullsized":"2024-04-28T12:49:35.810695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:52:24.574445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.639014+0000","last_change":"2024-04-28T12:48:44.485372+0000","last_active":"2024-04-28T12:49:35.639014+0000","last_peered":"2024-04-28T12:49:35.639014+0000","last_clean":"2024-04-28T12:49:35.639014+0000","last_became_active":"2024-04-28T12:48:44.485074+0000","last_became_peered":"2024-04-28T12:48:44.485074+0000","last_unstale":"2024-04-28T12:49:35.639014+0000","last_undegraded":"2024-04-28T12:49:35.639014+0000","last_fullsized":"2024-04-28T12:49:35.639014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:24:39.390059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638394+0000","last_change":"2024-04-28T12:49:35.638394+0000","last_active":"2024-04-28T12:49:35.638394+0000","last_peered":"2024-04-28T12:49:35.638394+0000","last_clean":"2024-04-28T12:49:35.638394+0000","last_became_active":"2024-04-28T12:49:35.637821+0000","last_became_peered":"2024-04-28T12:49:35.637821+0000","last_unstale":"2024-04-28T12:49:35.638394+0000","last_undegraded":"2024-04-28T12:49:35.638394+0000","last_fullsized":"2024-04-28T12:49:35.638394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:10:54.535365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631517+0000","last_change":"2024-04-28T12:48:44.479273+0000","last_active":"2024-04-28T12:49:35.631517+0000","last_peered":"2024-04-28T12:49:35.631517+0000","last_clean":"2024-04-28T12:49:35.631517+0000","last_became_active":"2024-04-28T12:48:44.478962+0000","last_became_peered":"2024-04-28T12:48:44.478962+0000","last_unstale":"2024-04-28T12:49:35.631517+0000","last_undegraded":"2024-04-28T12:49:35.631517+0000","last_fullsized":"2024-04-28T12:49:35.631517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:27.698365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074991+0000","last_change":"2024-04-28T12:48:44.478442+0000","last_active":"2024-04-28T12:49:36.074991+0000","last_peered":"2024-04-28T12:49:36.074991+0000","last_clean":"2024-04-28T12:49:36.074991+0000","last_became_active":"2024-04-28T12:48:44.478121+0000","last_became_peered":"2024-04-28T12:48:44.478121+0000","last_unstale":"2024-04-28T12:49:36.074991+0000","last_undegraded":"2024-04-28T12:49:36.074991+0000","last_fullsized":"2024-04-28T12:49:36.074991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:07:43.195804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074409+0000","last_change":"2024-04-28T12:48:44.476475+0000","last_active":"2024-04-28T12:49:36.074409+0000","last_peered":"2024-04-28T12:49:36.074409+0000","last_clean":"2024-04-28T12:49:36.074409+0000","last_became_active":"2024-04-28T12:48:44.475839+0000","last_became_peered":"2024-04-28T12:48:44.475839+0000","last_unstale":"2024-04-28T12:49:36.074409+0000","last_undegraded":"2024-04-28T12:49:36.074409+0000","last_fullsized":"2024-04-28T12:49:36.074409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:34:51.513356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630656+0000","last_change":"2024-04-28T12:48:44.480569+0000","last_active":"2024-04-28T12:49:35.630656+0000","last_peered":"2024-04-28T12:49:35.630656+0000","last_clean":"2024-04-28T12:49:35.630656+0000","last_became_active":"2024-04-28T12:48:44.478085+0000","last_became_peered":"2024-04-28T12:48:44.478085+0000","last_unstale":"2024-04-28T12:49:35.630656+0000","last_undegraded":"2024-04-28T12:49:35.630656+0000","last_fullsized":"2024-04-28T12:49:35.630656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:23:13.488078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.636591+0000","last_change":"2024-04-28T12:49:35.636591+0000","last_active":"2024-04-28T12:49:35.636591+0000","last_peered":"2024-04-28T12:49:35.636591+0000","last_clean":"2024-04-28T12:49:35.636591+0000","last_became_active":"2024-04-28T12:49:35.636212+0000","last_became_peered":"2024-04-28T12:49:35.636212+0000","last_unstale":"2024-04-28T12:49:35.636591+0000","last_undegraded":"2024-04-28T12:49:35.636591+0000","last_fullsized":"2024-04-28T12:49:35.636591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:49:13.868018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638020+0000","last_change":"2024-04-28T12:48:44.485600+0000","last_active":"2024-04-28T12:49:35.638020+0000","last_peered":"2024-04-28T12:49:35.638020+0000","last_clean":"2024-04-28T12:49:35.638020+0000","last_became_active":"2024-04-28T12:48:44.485297+0000","last_became_peered":"2024-04-28T12:48:44.485297+0000","last_unstale":"2024-04-28T12:49:35.638020+0000","last_undegraded":"2024-04-28T12:49:35.638020+0000","last_fullsized":"2024-04-28T12:49:35.638020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:25:16.124798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630967+0000","last_change":"2024-04-28T12:48:44.487376+0000","last_active":"2024-04-28T12:49:35.630967+0000","last_peered":"2024-04-28T12:49:35.630967+0000","last_clean":"2024-04-28T12:49:35.630967+0000","last_became_active":"2024-04-28T12:48:44.486561+0000","last_became_peered":"2024-04-28T12:48:44.486561+0000","last_unstale":"2024-04-28T12:49:35.630967+0000","last_undegraded":"2024-04-28T12:49:35.630967+0000","last_fullsized":"2024-04-28T12:49:35.630967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:43.398130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628490+0000","last_change":"2024-04-28T12:48:44.475470+0000","last_active":"2024-04-28T12:49:35.628490+0000","last_peered":"2024-04-28T12:49:35.628490+0000","last_clean":"2024-04-28T12:49:35.628490+0000","last_became_active":"2024-04-28T12:48:44.475113+0000","last_became_peered":"2024-04-28T12:48:44.475113+0000","last_unstale":"2024-04-28T12:49:35.628490+0000","last_undegraded":"2024-04-28T12:49:35.628490+0000","last_fullsized":"2024-04-28T12:49:35.628490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:55:28.688458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634384+0000","last_change":"2024-04-28T12:48:44.482195+0000","last_active":"2024-04-28T12:49:35.634384+0000","last_peered":"2024-04-28T12:49:35.634384+0000","last_clean":"2024-04-28T12:49:35.634384+0000","last_became_active":"2024-04-28T12:48:44.481847+0000","last_became_peered":"2024-04-28T12:48:44.481847+0000","last_unstale":"2024-04-28T12:49:35.634384+0000","last_undegraded":"2024-04-28T12:49:35.634384+0000","last_fullsized":"2024-04-28T12:49:35.634384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:18:03.507012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628309+0000","last_change":"2024-04-28T12:48:43.517808+0000","last_active":"2024-04-28T12:49:35.628309+0000","last_peered":"2024-04-28T12:49:35.628309+0000","last_clean":"2024-04-28T12:49:35.628309+0000","last_became_active":"2024-04-28T12:48:43.517443+0000","last_became_peered":"2024-04-28T12:48:43.517443+0000","last_unstale":"2024-04-28T12:49:35.628309+0000","last_undegraded":"2024-04-28T12:49:35.628309+0000","last_fullsized":"2024-04-28T12:49:35.628309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:57:11.972794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811121+0000","last_change":"2024-04-28T12:48:44.486296+0000","last_active":"2024-04-28T12:49:35.811121+0000","last_peered":"2024-04-28T12:49:35.811121+0000","last_clean":"2024-04-28T12:49:35.811121+0000","last_became_active":"2024-04-28T12:48:44.485289+0000","last_became_peered":"2024-04-28T12:48:44.485289+0000","last_unstale":"2024-04-28T12:49:35.811121+0000","last_undegraded":"2024-04-28T12:49:35.811121+0000","last_fullsized":"2024-04-28T12:49:35.811121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:07.558478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811400+0000","last_change":"2024-04-28T12:48:43.523725+0000","last_active":"2024-04-28T12:49:35.811400+0000","last_peered":"2024-04-28T12:49:35.811400+0000","last_clean":"2024-04-28T12:49:35.811400+0000","last_became_active":"2024-04-28T12:48:43.523367+0000","last_became_peered":"2024-04-28T12:48:43.523367+0000","last_unstale":"2024-04-28T12:49:35.811400+0000","last_undegraded":"2024-04-28T12:49:35.811400+0000","last_fullsized":"2024-04-28T12:49:35.811400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:15:15.674552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631210+0000","last_change":"2024-04-28T12:48:44.473393+0000","last_active":"2024-04-28T12:49:35.631210+0000","last_peered":"2024-04-28T12:49:35.631210+0000","last_clean":"2024-04-28T12:49:35.631210+0000","last_became_active":"2024-04-28T12:48:44.473035+0000","last_became_peered":"2024-04-28T12:48:44.473035+0000","last_unstale":"2024-04-28T12:49:35.631210+0000","last_undegraded":"2024-04-28T12:49:35.631210+0000","last_fullsized":"2024-04-28T12:49:35.631210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:06.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.647559+0000","last_change":"2024-04-28T12:49:35.647105+0000","last_active":"2024-04-28T12:49:35.647559+0000","last_peered":"2024-04-28T12:49:35.647559+0000","last_clean":"2024-04-28T12:49:35.647559+0000","last_became_active":"2024-04-28T12:49:35.636538+0000","last_became_peered":"2024-04-28T12:49:35.636538+0000","last_unstale":"2024-04-28T12:49:35.647559+0000","last_undegraded":"2024-04-28T12:49:35.647559+0000","last_fullsized":"2024-04-28T12:49:35.647559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:33:20.769651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638947+0000","last_change":"2024-04-28T12:48:44.475616+0000","last_active":"2024-04-28T12:49:35.638947+0000","last_peered":"2024-04-28T12:49:35.638947+0000","last_clean":"2024-04-28T12:49:35.638947+0000","last_became_active":"2024-04-28T12:48:44.475268+0000","last_became_peered":"2024-04-28T12:48:44.475268+0000","last_unstale":"2024-04-28T12:49:35.638947+0000","last_undegraded":"2024-04-28T12:49:35.638947+0000","last_fullsized":"2024-04-28T12:49:35.638947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:15:25.896749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635723+0000","last_change":"2024-04-28T12:48:43.517652+0000","last_active":"2024-04-28T12:49:35.635723+0000","last_peered":"2024-04-28T12:49:35.635723+0000","last_clean":"2024-04-28T12:49:35.635723+0000","last_became_active":"2024-04-28T12:48:43.517220+0000","last_became_peered":"2024-04-28T12:48:43.517220+0000","last_unstale":"2024-04-28T12:49:35.635723+0000","last_undegraded":"2024-04-28T12:49:35.635723+0000","last_fullsized":"2024-04-28T12:49:35.635723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:25:52.793542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075072+0000","last_change":"2024-04-28T12:48:44.488860+0000","last_active":"2024-04-28T12:49:36.075072+0000","last_peered":"2024-04-28T12:49:36.075072+0000","last_clean":"2024-04-28T12:49:36.075072+0000","last_became_active":"2024-04-28T12:48:44.488487+0000","last_became_peered":"2024-04-28T12:48:44.488487+0000","last_unstale":"2024-04-28T12:49:36.075072+0000","last_undegraded":"2024-04-28T12:49:36.075072+0000","last_fullsized":"2024-04-28T12:49:36.075072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:33:28.357280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:57.558809+0000","last_change":"2024-04-28T12:48:43.507961+0000","last_active":"2024-04-28T12:49:57.558809+0000","last_peered":"2024-04-28T12:49:57.558809+0000","last_clean":"2024-04-28T12:49:57.558809+0000","last_became_active":"2024-04-28T12:48:43.507600+0000","last_became_peered":"2024-04-28T12:48:43.507600+0000","last_unstale":"2024-04-28T12:49:57.558809+0000","last_undegraded":"2024-04-28T12:49:57.558809+0000","last_fullsized":"2024-04-28T12:49:57.558809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:16.188154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634653+0000","last_change":"2024-04-28T12:48:44.474027+0000","last_active":"2024-04-28T12:49:35.634653+0000","last_peered":"2024-04-28T12:49:35.634653+0000","last_clean":"2024-04-28T12:49:35.634653+0000","last_became_active":"2024-04-28T12:48:44.473372+0000","last_became_peered":"2024-04-28T12:48:44.473372+0000","last_unstale":"2024-04-28T12:49:35.634653+0000","last_undegraded":"2024-04-28T12:49:35.634653+0000","last_fullsized":"2024-04-28T12:49:35.634653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:18:43.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634657+0000","last_change":"2024-04-28T12:48:43.521135+0000","last_active":"2024-04-28T12:49:35.634657+0000","last_peered":"2024-04-28T12:49:35.634657+0000","last_clean":"2024-04-28T12:49:35.634657+0000","last_became_active":"2024-04-28T12:48:43.520749+0000","last_became_peered":"2024-04-28T12:48:43.520749+0000","last_unstale":"2024-04-28T12:49:35.634657+0000","last_undegraded":"2024-04-28T12:49:35.634657+0000","last_fullsized":"2024-04-28T12:49:35.634657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:35:33.545386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074946+0000","last_change":"2024-04-28T12:48:44.479090+0000","last_active":"2024-04-28T12:49:36.074946+0000","last_peered":"2024-04-28T12:49:36.074946+0000","last_clean":"2024-04-28T12:49:36.074946+0000","last_became_active":"2024-04-28T12:48:44.478725+0000","last_became_peered":"2024-04-28T12:48:44.478725+0000","last_unstale":"2024-04-28T12:49:36.074946+0000","last_undegraded":"2024-04-28T12:49:36.074946+0000","last_fullsized":"2024-04-28T12:49:36.074946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:40:21.710990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074274+0000","last_change":"2024-04-28T12:48:43.505309+0000","last_active":"2024-04-28T12:49:36.074274+0000","last_peered":"2024-04-28T12:49:36.074274+0000","last_clean":"2024-04-28T12:49:36.074274+0000","last_became_active":"2024-04-28T12:48:43.504934+0000","last_became_peered":"2024-04-28T12:48:43.504934+0000","last_unstale":"2024-04-28T12:49:36.074274+0000","last_undegraded":"2024-04-28T12:49:36.074274+0000","last_fullsized":"2024-04-28T12:49:36.074274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:48:39.652606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810767+0000","last_change":"2024-04-28T12:48:44.479930+0000","last_active":"2024-04-28T12:49:35.810767+0000","last_peered":"2024-04-28T12:49:35.810767+0000","last_clean":"2024-04-28T12:49:35.810767+0000","last_became_active":"2024-04-28T12:48:44.479571+0000","last_became_peered":"2024-04-28T12:48:44.479571+0000","last_unstale":"2024-04-28T12:49:35.810767+0000","last_undegraded":"2024-04-28T12:49:35.810767+0000","last_fullsized":"2024-04-28T12:49:35.810767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:00.130664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074739+0000","last_change":"2024-04-28T12:48:43.508935+0000","last_active":"2024-04-28T12:49:36.074739+0000","last_peered":"2024-04-28T12:49:36.074739+0000","last_clean":"2024-04-28T12:49:36.074739+0000","last_became_active":"2024-04-28T12:48:43.508596+0000","last_became_peered":"2024-04-28T12:48:43.508596+0000","last_unstale":"2024-04-28T12:49:36.074739+0000","last_undegraded":"2024-04-28T12:49:36.074739+0000","last_fullsized":"2024-04-28T12:49:36.074739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:41.905528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628233+0000","last_change":"2024-04-28T12:48:44.489695+0000","last_active":"2024-04-28T12:49:35.628233+0000","last_peered":"2024-04-28T12:49:35.628233+0000","last_clean":"2024-04-28T12:49:35.628233+0000","last_became_active":"2024-04-28T12:48:44.489290+0000","last_became_peered":"2024-04-28T12:48:44.489290+0000","last_unstale":"2024-04-28T12:49:35.628233+0000","last_undegraded":"2024-04-28T12:49:35.628233+0000","last_fullsized":"2024-04-28T12:49:35.628233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:34:01.873775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":92,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:57.558319+0000","last_change":"2024-04-28T12:48:43.511751+0000","last_active":"2024-04-28T12:49:57.558319+0000","last_peered":"2024-04-28T12:49:57.558319+0000","last_clean":"2024-04-28T12:49:57.558319+0000","last_became_active":"2024-04-28T12:48:43.511428+0000","last_became_peered":"2024-04-28T12:48:43.511428+0000","last_unstale":"2024-04-28T12:49:57.558319+0000","last_undegraded":"2024-04-28T12:49:57.558319+0000","last_fullsized":"2024-04-28T12:49:57.558319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:12.987697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.639163+0000","last_change":"2024-04-28T12:48:44.472310+0000","last_active":"2024-04-28T12:49:35.639163+0000","last_peered":"2024-04-28T12:49:35.639163+0000","last_clean":"2024-04-28T12:49:35.639163+0000","last_became_active":"2024-04-28T12:48:44.471978+0000","last_became_peered":"2024-04-28T12:48:44.471978+0000","last_unstale":"2024-04-28T12:49:35.639163+0000","last_undegraded":"2024-04-28T12:49:35.639163+0000","last_fullsized":"2024-04-28T12:49:35.639163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.068156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631642+0000","last_change":"2024-04-28T12:48:43.514475+0000","last_active":"2024-04-28T12:49:35.631642+0000","last_peered":"2024-04-28T12:49:35.631642+0000","last_clean":"2024-04-28T12:49:35.631642+0000","last_became_active":"2024-04-28T12:48:43.514067+0000","last_became_peered":"2024-04-28T12:48:43.514067+0000","last_unstale":"2024-04-28T12:49:35.631642+0000","last_undegraded":"2024-04-28T12:49:35.631642+0000","last_fullsized":"2024-04-28T12:49:35.631642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:52.125456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631188+0000","last_change":"2024-04-28T12:48:44.479210+0000","last_active":"2024-04-28T12:49:35.631188+0000","last_peered":"2024-04-28T12:49:35.631188+0000","last_clean":"2024-04-28T12:49:35.631188+0000","last_became_active":"2024-04-28T12:48:44.478816+0000","last_became_peered":"2024-04-28T12:48:44.478816+0000","last_unstale":"2024-04-28T12:49:35.631188+0000","last_undegraded":"2024-04-28T12:49:35.631188+0000","last_fullsized":"2024-04-28T12:49:35.631188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:03:06.994063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075237+0000","last_change":"2024-04-28T12:48:43.520207+0000","last_active":"2024-04-28T12:49:36.075237+0000","last_peered":"2024-04-28T12:49:36.075237+0000","last_clean":"2024-04-28T12:49:36.075237+0000","last_became_active":"2024-04-28T12:48:43.519823+0000","last_became_peered":"2024-04-28T12:48:43.519823+0000","last_unstale":"2024-04-28T12:49:36.075237+0000","last_undegraded":"2024-04-28T12:49:36.075237+0000","last_fullsized":"2024-04-28T12:49:36.075237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:02.238166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638709+0000","last_change":"2024-04-28T12:48:44.479632+0000","last_active":"2024-04-28T12:49:35.638709+0000","last_peered":"2024-04-28T12:49:35.638709+0000","last_clean":"2024-04-28T12:49:35.638709+0000","last_became_active":"2024-04-28T12:48:44.479197+0000","last_became_peered":"2024-04-28T12:48:44.479197+0000","last_unstale":"2024-04-28T12:49:35.638709+0000","last_undegraded":"2024-04-28T12:49:35.638709+0000","last_fullsized":"2024-04-28T12:49:35.638709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:49:01.557312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'25","reported_seq":208,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:59.116196+0000","last_change":"2024-04-28T12:49:36.083466+0000","last_active":"2024-04-28T12:49:59.116196+0000","last_peered":"2024-04-28T12:49:59.116196+0000","last_clean":"2024-04-28T12:49:59.116196+0000","last_became_active":"2024-04-28T12:49:36.076359+0000","last_became_peered":"2024-04-28T12:49:36.076359+0000","last_unstale":"2024-04-28T12:49:59.116196+0000","last_undegraded":"2024-04-28T12:49:59.116196+0000","last_fullsized":"2024-04-28T12:49:59.116196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:08.982295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50398,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":138,"num_read_kb":117,"num_write":26,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":50398,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635960+0000","last_change":"2024-04-28T12:48:44.484996+0000","last_active":"2024-04-28T12:49:35.635960+0000","last_peered":"2024-04-28T12:49:35.635960+0000","last_clean":"2024-04-28T12:49:35.635960+0000","last_became_active":"2024-04-28T12:48:44.484591+0000","last_became_peered":"2024-04-28T12:48:44.484591+0000","last_unstale":"2024-04-28T12:49:35.635960+0000","last_undegraded":"2024-04-28T12:49:35.635960+0000","last_fullsized":"2024-04-28T12:49:35.635960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:38:42.768757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633319+0000","last_change":"2024-04-28T12:48:43.510365+0000","last_active":"2024-04-28T12:49:35.633319+0000","last_peered":"2024-04-28T12:49:35.633319+0000","last_clean":"2024-04-28T12:49:35.633319+0000","last_became_active":"2024-04-28T12:48:43.510028+0000","last_became_peered":"2024-04-28T12:48:43.510028+0000","last_unstale":"2024-04-28T12:49:35.633319+0000","last_undegraded":"2024-04-28T12:49:35.633319+0000","last_fullsized":"2024-04-28T12:49:35.633319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:09:20.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074327+0000","last_change":"2024-04-28T12:48:44.489704+0000","last_active":"2024-04-28T12:49:36.074327+0000","last_peered":"2024-04-28T12:49:36.074327+0000","last_clean":"2024-04-28T12:49:36.074327+0000","last_became_active":"2024-04-28T12:48:44.489331+0000","last_became_peered":"2024-04-28T12:48:44.489331+0000","last_unstale":"2024-04-28T12:49:36.074327+0000","last_undegraded":"2024-04-28T12:49:36.074327+0000","last_fullsized":"2024-04-28T12:49:36.074327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:33.755818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074743+0000","last_change":"2024-04-28T12:48:43.519213+0000","last_active":"2024-04-28T12:49:36.074743+0000","last_peered":"2024-04-28T12:49:36.074743+0000","last_clean":"2024-04-28T12:49:36.074743+0000","last_became_active":"2024-04-28T12:48:43.518385+0000","last_became_peered":"2024-04-28T12:48:43.518385+0000","last_unstale":"2024-04-28T12:49:36.074743+0000","last_undegraded":"2024-04-28T12:49:36.074743+0000","last_fullsized":"2024-04-28T12:49:36.074743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:31:37.130504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631449+0000","last_change":"2024-04-28T12:48:44.480838+0000","last_active":"2024-04-28T12:49:35.631449+0000","last_peered":"2024-04-28T12:49:35.631449+0000","last_clean":"2024-04-28T12:49:35.631449+0000","last_became_active":"2024-04-28T12:48:44.480522+0000","last_became_peered":"2024-04-28T12:48:44.480522+0000","last_unstale":"2024-04-28T12:49:35.631449+0000","last_undegraded":"2024-04-28T12:49:35.631449+0000","last_fullsized":"2024-04-28T12:49:35.631449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:24.470154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:58.559833+0000","last_change":"2024-04-28T12:48:43.505498+0000","last_active":"2024-04-28T12:49:58.559833+0000","last_peered":"2024-04-28T12:49:58.559833+0000","last_clean":"2024-04-28T12:49:58.559833+0000","last_became_active":"2024-04-28T12:48:43.505041+0000","last_became_peered":"2024-04-28T12:48:43.505041+0000","last_unstale":"2024-04-28T12:49:58.559833+0000","last_undegraded":"2024-04-28T12:49:58.559833+0000","last_fullsized":"2024-04-28T12:49:58.559833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:36.624508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631059+0000","last_change":"2024-04-28T12:48:44.482698+0000","last_active":"2024-04-28T12:49:35.631059+0000","last_peered":"2024-04-28T12:49:35.631059+0000","last_clean":"2024-04-28T12:49:35.631059+0000","last_became_active":"2024-04-28T12:48:44.482365+0000","last_became_peered":"2024-04-28T12:48:44.482365+0000","last_unstale":"2024-04-28T12:49:35.631059+0000","last_undegraded":"2024-04-28T12:49:35.631059+0000","last_fullsized":"2024-04-28T12:49:35.631059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:45:33.648105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634552+0000","last_change":"2024-04-28T12:48:43.515942+0000","last_active":"2024-04-28T12:49:35.634552+0000","last_peered":"2024-04-28T12:49:35.634552+0000","last_clean":"2024-04-28T12:49:35.634552+0000","last_became_active":"2024-04-28T12:48:43.515497+0000","last_became_peered":"2024-04-28T12:48:43.515497+0000","last_unstale":"2024-04-28T12:49:35.634552+0000","last_undegraded":"2024-04-28T12:49:35.634552+0000","last_fullsized":"2024-04-28T12:49:35.634552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:39.489486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074619+0000","last_change":"2024-04-28T12:48:44.487831+0000","last_active":"2024-04-28T12:49:36.074619+0000","last_peered":"2024-04-28T12:49:36.074619+0000","last_clean":"2024-04-28T12:49:36.074619+0000","last_became_active":"2024-04-28T12:48:44.487341+0000","last_became_peered":"2024-04-28T12:48:44.487341+0000","last_unstale":"2024-04-28T12:49:36.074619+0000","last_undegraded":"2024-04-28T12:49:36.074619+0000","last_fullsized":"2024-04-28T12:49:36.074619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:42.810514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635784+0000","last_change":"2024-04-28T12:48:43.513667+0000","last_active":"2024-04-28T12:49:35.635784+0000","last_peered":"2024-04-28T12:49:35.635784+0000","last_clean":"2024-04-28T12:49:35.635784+0000","last_became_active":"2024-04-28T12:48:43.513192+0000","last_became_peered":"2024-04-28T12:48:43.513192+0000","last_unstale":"2024-04-28T12:49:35.635784+0000","last_undegraded":"2024-04-28T12:49:35.635784+0000","last_fullsized":"2024-04-28T12:49:35.635784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.470320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630764+0000","last_change":"2024-04-28T12:48:44.478203+0000","last_active":"2024-04-28T12:49:35.630764+0000","last_peered":"2024-04-28T12:49:35.630764+0000","last_clean":"2024-04-28T12:49:35.630764+0000","last_became_active":"2024-04-28T12:48:44.476238+0000","last_became_peered":"2024-04-28T12:48:44.476238+0000","last_unstale":"2024-04-28T12:49:35.630764+0000","last_undegraded":"2024-04-28T12:49:35.630764+0000","last_fullsized":"2024-04-28T12:49:35.630764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:46:49.810015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628368+0000","last_change":"2024-04-28T12:48:43.516036+0000","last_active":"2024-04-28T12:49:35.628368+0000","last_peered":"2024-04-28T12:49:35.628368+0000","last_clean":"2024-04-28T12:49:35.628368+0000","last_became_active":"2024-04-28T12:48:43.515494+0000","last_became_peered":"2024-04-28T12:48:43.515494+0000","last_unstale":"2024-04-28T12:49:35.628368+0000","last_undegraded":"2024-04-28T12:49:35.628368+0000","last_fullsized":"2024-04-28T12:49:35.628368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:34:56.568179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634892+0000","last_change":"2024-04-28T12:48:44.476836+0000","last_active":"2024-04-28T12:49:35.634892+0000","last_peered":"2024-04-28T12:49:35.634892+0000","last_clean":"2024-04-28T12:49:35.634892+0000","last_became_active":"2024-04-28T12:48:44.476214+0000","last_became_peered":"2024-04-28T12:48:44.476214+0000","last_unstale":"2024-04-28T12:49:35.634892+0000","last_undegraded":"2024-04-28T12:49:35.634892+0000","last_fullsized":"2024-04-28T12:49:35.634892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:24:26.707985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631213+0000","last_change":"2024-04-28T12:48:43.520918+0000","last_active":"2024-04-28T12:49:35.631213+0000","last_peered":"2024-04-28T12:49:35.631213+0000","last_clean":"2024-04-28T12:49:35.631213+0000","last_became_active":"2024-04-28T12:48:43.520564+0000","last_became_peered":"2024-04-28T12:48:43.520564+0000","last_unstale":"2024-04-28T12:49:35.631213+0000","last_undegraded":"2024-04-28T12:49:35.631213+0000","last_fullsized":"2024-04-28T12:49:35.631213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:32:51.206102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635059+0000","last_change":"2024-04-28T12:48:44.484570+0000","last_active":"2024-04-28T12:49:35.635059+0000","last_peered":"2024-04-28T12:49:35.635059+0000","last_clean":"2024-04-28T12:49:35.635059+0000","last_became_active":"2024-04-28T12:48:44.484133+0000","last_became_peered":"2024-04-28T12:48:44.484133+0000","last_unstale":"2024-04-28T12:49:35.635059+0000","last_undegraded":"2024-04-28T12:49:35.635059+0000","last_fullsized":"2024-04-28T12:49:35.635059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:59:14.274873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074978+0000","last_change":"2024-04-28T12:48:44.478013+0000","last_active":"2024-04-28T12:49:36.074978+0000","last_peered":"2024-04-28T12:49:36.074978+0000","last_clean":"2024-04-28T12:49:36.074978+0000","last_became_active":"2024-04-28T12:48:44.477559+0000","last_became_peered":"2024-04-28T12:48:44.477559+0000","last_unstale":"2024-04-28T12:49:36.074978+0000","last_undegraded":"2024-04-28T12:49:36.074978+0000","last_fullsized":"2024-04-28T12:49:36.074978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:57:49.931865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:59.560432+0000","last_change":"2024-04-28T12:48:43.515873+0000","last_active":"2024-04-28T12:49:59.560432+0000","last_peered":"2024-04-28T12:49:59.560432+0000","last_clean":"2024-04-28T12:49:59.560432+0000","last_became_active":"2024-04-28T12:48:43.515442+0000","last_became_peered":"2024-04-28T12:48:43.515442+0000","last_unstale":"2024-04-28T12:49:59.560432+0000","last_undegraded":"2024-04-28T12:49:59.560432+0000","last_fullsized":"2024-04-28T12:49:59.560432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:29:57.973433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811001+0000","last_change":"2024-04-28T12:48:44.479868+0000","last_active":"2024-04-28T12:49:35.811001+0000","last_peered":"2024-04-28T12:49:35.811001+0000","last_clean":"2024-04-28T12:49:35.811001+0000","last_became_active":"2024-04-28T12:48:44.479519+0000","last_became_peered":"2024-04-28T12:48:44.479519+0000","last_unstale":"2024-04-28T12:49:35.811001+0000","last_undegraded":"2024-04-28T12:49:35.811001+0000","last_fullsized":"2024-04-28T12:49:35.811001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:09:20.946380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074568+0000","last_change":"2024-04-28T12:48:43.507662+0000","last_active":"2024-04-28T12:49:36.074568+0000","last_peered":"2024-04-28T12:49:36.074568+0000","last_clean":"2024-04-28T12:49:36.074568+0000","last_became_active":"2024-04-28T12:48:43.507331+0000","last_became_peered":"2024-04-28T12:48:43.507331+0000","last_unstale":"2024-04-28T12:49:36.074568+0000","last_undegraded":"2024-04-28T12:49:36.074568+0000","last_fullsized":"2024-04-28T12:49:36.074568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:10.073569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":342,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631169+0000","last_change":"2024-04-28T12:47:27.444407+0000","last_active":"2024-04-28T12:49:35.631169+0000","last_peered":"2024-04-28T12:49:35.631169+0000","last_clean":"2024-04-28T12:49:35.631169+0000","last_became_active":"2024-04-28T12:47:27.443992+0000","last_became_peered":"2024-04-28T12:47:27.443992+0000","last_unstale":"2024-04-28T12:49:35.631169+0000","last_undegraded":"2024-04-28T12:49:35.631169+0000","last_fullsized":"2024-04-28T12:49:35.631169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:44:00.636011+0000","last_clean_scrub_stamp":"2024-04-28T12:44:00.636011+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:17:23.614524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074164+0000","last_change":"2024-04-28T12:48:44.479395+0000","last_active":"2024-04-28T12:49:36.074164+0000","last_peered":"2024-04-28T12:49:36.074164+0000","last_clean":"2024-04-28T12:49:36.074164+0000","last_became_active":"2024-04-28T12:48:44.478935+0000","last_became_peered":"2024-04-28T12:48:44.478935+0000","last_unstale":"2024-04-28T12:49:36.074164+0000","last_undegraded":"2024-04-28T12:49:36.074164+0000","last_fullsized":"2024-04-28T12:49:36.074164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:19:28.212744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631257+0000","last_change":"2024-04-28T12:48:43.512026+0000","last_active":"2024-04-28T12:49:35.631257+0000","last_peered":"2024-04-28T12:49:35.631257+0000","last_clean":"2024-04-28T12:49:35.631257+0000","last_became_active":"2024-04-28T12:48:43.511650+0000","last_became_peered":"2024-04-28T12:48:43.511650+0000","last_unstale":"2024-04-28T12:49:35.631257+0000","last_undegraded":"2024-04-28T12:49:35.631257+0000","last_fullsized":"2024-04-28T12:49:35.631257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:38:04.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.641820+0000","last_change":"2024-04-28T12:49:35.641378+0000","last_active":"2024-04-28T12:49:35.641820+0000","last_peered":"2024-04-28T12:49:35.641820+0000","last_clean":"2024-04-28T12:49:35.641820+0000","last_became_active":"2024-04-28T12:49:35.633729+0000","last_became_peered":"2024-04-28T12:49:35.633729+0000","last_unstale":"2024-04-28T12:49:35.641820+0000","last_undegraded":"2024-04-28T12:49:35.641820+0000","last_fullsized":"2024-04-28T12:49:35.641820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:16:22.142685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074413+0000","last_change":"2024-04-28T12:48:44.470458+0000","last_active":"2024-04-28T12:49:36.074413+0000","last_peered":"2024-04-28T12:49:36.074413+0000","last_clean":"2024-04-28T12:49:36.074413+0000","last_became_active":"2024-04-28T12:48:44.470105+0000","last_became_peered":"2024-04-28T12:48:44.470105+0000","last_unstale":"2024-04-28T12:49:36.074413+0000","last_undegraded":"2024-04-28T12:49:36.074413+0000","last_fullsized":"2024-04-28T12:49:36.074413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:49:25.970990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.628468+0000","last_change":"2024-04-28T12:48:43.510065+0000","last_active":"2024-04-28T12:49:35.628468+0000","last_peered":"2024-04-28T12:49:35.628468+0000","last_clean":"2024-04-28T12:49:35.628468+0000","last_became_active":"2024-04-28T12:48:43.509718+0000","last_became_peered":"2024-04-28T12:48:43.509718+0000","last_unstale":"2024-04-28T12:49:35.628468+0000","last_undegraded":"2024-04-28T12:49:35.628468+0000","last_fullsized":"2024-04-28T12:49:35.628468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:50:46.776100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811200+0000","last_change":"2024-04-28T12:48:44.487022+0000","last_active":"2024-04-28T12:49:35.811200+0000","last_peered":"2024-04-28T12:49:35.811200+0000","last_clean":"2024-04-28T12:49:35.811200+0000","last_became_active":"2024-04-28T12:48:44.486675+0000","last_became_peered":"2024-04-28T12:48:44.486675+0000","last_unstale":"2024-04-28T12:49:35.811200+0000","last_undegraded":"2024-04-28T12:49:35.811200+0000","last_fullsized":"2024-04-28T12:49:35.811200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:36.185845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075014+0000","last_change":"2024-04-28T12:48:43.505464+0000","last_active":"2024-04-28T12:49:36.075014+0000","last_peered":"2024-04-28T12:49:36.075014+0000","last_clean":"2024-04-28T12:49:36.075014+0000","last_became_active":"2024-04-28T12:48:43.505085+0000","last_became_peered":"2024-04-28T12:48:43.505085+0000","last_unstale":"2024-04-28T12:49:36.075014+0000","last_undegraded":"2024-04-28T12:49:36.075014+0000","last_fullsized":"2024-04-28T12:49:36.075014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:37:47.875243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.812460+0000","last_change":"2024-04-28T12:49:35.812460+0000","last_active":"2024-04-28T12:49:35.812460+0000","last_peered":"2024-04-28T12:49:35.812460+0000","last_clean":"2024-04-28T12:49:35.812460+0000","last_became_active":"2024-04-28T12:49:35.812047+0000","last_became_peered":"2024-04-28T12:49:35.812047+0000","last_unstale":"2024-04-28T12:49:35.812460+0000","last_undegraded":"2024-04-28T12:49:35.812460+0000","last_fullsized":"2024-04-28T12:49:35.812460+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:22:47.979802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638144+0000","last_change":"2024-04-28T12:48:43.508992+0000","last_active":"2024-04-28T12:49:35.638144+0000","last_peered":"2024-04-28T12:49:35.638144+0000","last_clean":"2024-04-28T12:49:35.638144+0000","last_became_active":"2024-04-28T12:48:43.508492+0000","last_became_peered":"2024-04-28T12:48:43.508492+0000","last_unstale":"2024-04-28T12:49:35.638144+0000","last_undegraded":"2024-04-28T12:49:35.638144+0000","last_fullsized":"2024-04-28T12:49:35.638144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:56:29.138232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634927+0000","last_change":"2024-04-28T12:48:44.476462+0000","last_active":"2024-04-28T12:49:35.634927+0000","last_peered":"2024-04-28T12:49:35.634927+0000","last_clean":"2024-04-28T12:49:35.634927+0000","last_became_active":"2024-04-28T12:48:44.475660+0000","last_became_peered":"2024-04-28T12:48:44.475660+0000","last_unstale":"2024-04-28T12:49:35.634927+0000","last_undegraded":"2024-04-28T12:49:35.634927+0000","last_fullsized":"2024-04-28T12:49:35.634927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:49:48.656837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810825+0000","last_change":"2024-04-28T12:48:43.508946+0000","last_active":"2024-04-28T12:49:35.810825+0000","last_peered":"2024-04-28T12:49:35.810825+0000","last_clean":"2024-04-28T12:49:35.810825+0000","last_became_active":"2024-04-28T12:48:43.508553+0000","last_became_peered":"2024-04-28T12:48:43.508553+0000","last_unstale":"2024-04-28T12:49:35.810825+0000","last_undegraded":"2024-04-28T12:49:35.810825+0000","last_fullsized":"2024-04-28T12:49:35.810825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:52:56.461509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630544+0000","last_change":"2024-04-28T12:48:44.473967+0000","last_active":"2024-04-28T12:49:35.630544+0000","last_peered":"2024-04-28T12:49:35.630544+0000","last_clean":"2024-04-28T12:49:35.630544+0000","last_became_active":"2024-04-28T12:48:44.473614+0000","last_became_peered":"2024-04-28T12:48:44.473614+0000","last_unstale":"2024-04-28T12:49:35.630544+0000","last_undegraded":"2024-04-28T12:49:35.630544+0000","last_fullsized":"2024-04-28T12:49:35.630544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:58:20.994924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633162+0000","last_change":"2024-04-28T12:48:43.511287+0000","last_active":"2024-04-28T12:49:35.633162+0000","last_peered":"2024-04-28T12:49:35.633162+0000","last_clean":"2024-04-28T12:49:35.633162+0000","last_became_active":"2024-04-28T12:48:43.510955+0000","last_became_peered":"2024-04-28T12:48:43.510955+0000","last_unstale":"2024-04-28T12:49:35.633162+0000","last_undegraded":"2024-04-28T12:49:35.633162+0000","last_fullsized":"2024-04-28T12:49:35.633162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:59:40.952405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811070+0000","last_change":"2024-04-28T12:48:44.480514+0000","last_active":"2024-04-28T12:49:35.811070+0000","last_peered":"2024-04-28T12:49:35.811070+0000","last_clean":"2024-04-28T12:49:35.811070+0000","last_became_active":"2024-04-28T12:48:44.480160+0000","last_became_peered":"2024-04-28T12:48:44.480160+0000","last_unstale":"2024-04-28T12:49:35.811070+0000","last_undegraded":"2024-04-28T12:49:35.811070+0000","last_fullsized":"2024-04-28T12:49:35.811070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:07:02.221492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.643739+0000","last_change":"2024-04-28T12:49:35.643739+0000","last_active":"2024-04-28T12:49:35.643739+0000","last_peered":"2024-04-28T12:49:35.643739+0000","last_clean":"2024-04-28T12:49:35.643739+0000","last_became_active":"2024-04-28T12:49:35.639497+0000","last_became_peered":"2024-04-28T12:49:35.639497+0000","last_unstale":"2024-04-28T12:49:35.643739+0000","last_undegraded":"2024-04-28T12:49:35.643739+0000","last_fullsized":"2024-04-28T12:49:35.643739+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:59:52.886324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631343+0000","last_change":"2024-04-28T12:48:44.474095+0000","last_active":"2024-04-28T12:49:35.631343+0000","last_peered":"2024-04-28T12:49:35.631343+0000","last_clean":"2024-04-28T12:49:35.631343+0000","last_became_active":"2024-04-28T12:48:44.473377+0000","last_became_peered":"2024-04-28T12:48:44.473377+0000","last_unstale":"2024-04-28T12:49:35.631343+0000","last_undegraded":"2024-04-28T12:49:35.631343+0000","last_fullsized":"2024-04-28T12:49:35.631343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:29:29.355527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.634193+0000","last_change":"2024-04-28T12:48:43.515654+0000","last_active":"2024-04-28T12:49:35.634193+0000","last_peered":"2024-04-28T12:49:35.634193+0000","last_clean":"2024-04-28T12:49:35.634193+0000","last_became_active":"2024-04-28T12:48:43.515056+0000","last_became_peered":"2024-04-28T12:48:43.515056+0000","last_unstale":"2024-04-28T12:49:35.634193+0000","last_undegraded":"2024-04-28T12:49:35.634193+0000","last_fullsized":"2024-04-28T12:49:35.634193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:47:58.297248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631641+0000","last_change":"2024-04-28T12:48:44.474106+0000","last_active":"2024-04-28T12:49:35.631641+0000","last_peered":"2024-04-28T12:49:35.631641+0000","last_clean":"2024-04-28T12:49:35.631641+0000","last_became_active":"2024-04-28T12:48:44.473625+0000","last_became_peered":"2024-04-28T12:48:44.473625+0000","last_unstale":"2024-04-28T12:49:35.631641+0000","last_undegraded":"2024-04-28T12:49:35.631641+0000","last_fullsized":"2024-04-28T12:49:35.631641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:28:44.434155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.075075+0000","last_change":"2024-04-28T12:48:43.519658+0000","last_active":"2024-04-28T12:49:36.075075+0000","last_peered":"2024-04-28T12:49:36.075075+0000","last_clean":"2024-04-28T12:49:36.075075+0000","last_became_active":"2024-04-28T12:48:43.519291+0000","last_became_peered":"2024-04-28T12:48:43.519291+0000","last_unstale":"2024-04-28T12:49:36.075075+0000","last_undegraded":"2024-04-28T12:49:36.075075+0000","last_fullsized":"2024-04-28T12:49:36.075075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:41.356834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631413+0000","last_change":"2024-04-28T12:48:44.483790+0000","last_active":"2024-04-28T12:49:35.631413+0000","last_peered":"2024-04-28T12:49:35.631413+0000","last_clean":"2024-04-28T12:49:35.631413+0000","last_became_active":"2024-04-28T12:48:44.483442+0000","last_became_peered":"2024-04-28T12:48:44.483442+0000","last_unstale":"2024-04-28T12:49:35.631413+0000","last_undegraded":"2024-04-28T12:49:35.631413+0000","last_fullsized":"2024-04-28T12:49:35.631413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:38:59.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074428+0000","last_change":"2024-04-28T12:48:43.505650+0000","last_active":"2024-04-28T12:49:36.074428+0000","last_peered":"2024-04-28T12:49:36.074428+0000","last_clean":"2024-04-28T12:49:36.074428+0000","last_became_active":"2024-04-28T12:48:43.505305+0000","last_became_peered":"2024-04-28T12:48:43.505305+0000","last_unstale":"2024-04-28T12:49:36.074428+0000","last_undegraded":"2024-04-28T12:49:36.074428+0000","last_fullsized":"2024-04-28T12:49:36.074428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:42.485733+0000","last_clean_scrub_stamp":"2024-04-28T12:48:42.485733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:56:34.654105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631739+0000","last_change":"2024-04-28T12:48:44.478335+0000","last_active":"2024-04-28T12:49:35.631739+0000","last_peered":"2024-04-28T12:49:35.631739+0000","last_clean":"2024-04-28T12:49:35.631739+0000","last_became_active":"2024-04-28T12:48:44.477958+0000","last_became_peered":"2024-04-28T12:48:44.477958+0000","last_unstale":"2024-04-28T12:49:35.631739+0000","last_undegraded":"2024-04-28T12:49:35.631739+0000","last_fullsized":"2024-04-28T12:49:35.631739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:27.168011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.637326+0000","last_change":"2024-04-28T12:49:35.637326+0000","last_active":"2024-04-28T12:49:35.637326+0000","last_peered":"2024-04-28T12:49:35.637326+0000","last_clean":"2024-04-28T12:49:35.637326+0000","last_became_active":"2024-04-28T12:49:35.636959+0000","last_became_peered":"2024-04-28T12:49:35.636959+0000","last_unstale":"2024-04-28T12:49:35.637326+0000","last_undegraded":"2024-04-28T12:49:35.637326+0000","last_fullsized":"2024-04-28T12:49:35.637326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:40:22.428732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633418+0000","last_change":"2024-04-28T12:48:44.478553+0000","last_active":"2024-04-28T12:49:35.633418+0000","last_peered":"2024-04-28T12:49:35.633418+0000","last_clean":"2024-04-28T12:49:35.633418+0000","last_became_active":"2024-04-28T12:48:44.477539+0000","last_became_peered":"2024-04-28T12:48:44.477539+0000","last_unstale":"2024-04-28T12:49:35.633418+0000","last_undegraded":"2024-04-28T12:49:35.633418+0000","last_fullsized":"2024-04-28T12:49:35.633418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:03:44.149337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.635856+0000","last_change":"2024-04-28T12:49:35.635856+0000","last_active":"2024-04-28T12:49:35.635856+0000","last_peered":"2024-04-28T12:49:35.635856+0000","last_clean":"2024-04-28T12:49:35.635856+0000","last_became_active":"2024-04-28T12:49:35.635343+0000","last_became_peered":"2024-04-28T12:49:35.635343+0000","last_unstale":"2024-04-28T12:49:35.635856+0000","last_undegraded":"2024-04-28T12:49:35.635856+0000","last_fullsized":"2024-04-28T12:49:35.635856+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:32:09.319536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630843+0000","last_change":"2024-04-28T12:48:44.479729+0000","last_active":"2024-04-28T12:49:35.630843+0000","last_peered":"2024-04-28T12:49:35.630843+0000","last_clean":"2024-04-28T12:49:35.630843+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:49:35.630843+0000","last_undegraded":"2024-04-28T12:49:35.630843+0000","last_fullsized":"2024-04-28T12:49:35.630843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:33:32.456549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074522+0000","last_change":"2024-04-28T12:48:44.486785+0000","last_active":"2024-04-28T12:49:36.074522+0000","last_peered":"2024-04-28T12:49:36.074522+0000","last_clean":"2024-04-28T12:49:36.074522+0000","last_became_active":"2024-04-28T12:48:44.485634+0000","last_became_peered":"2024-04-28T12:48:44.485634+0000","last_unstale":"2024-04-28T12:49:36.074522+0000","last_undegraded":"2024-04-28T12:49:36.074522+0000","last_fullsized":"2024-04-28T12:49:36.074522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:58.148545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.631237+0000","last_change":"2024-04-28T12:48:44.479690+0000","last_active":"2024-04-28T12:49:35.631237+0000","last_peered":"2024-04-28T12:49:35.631237+0000","last_clean":"2024-04-28T12:49:35.631237+0000","last_became_active":"2024-04-28T12:48:44.479323+0000","last_became_peered":"2024-04-28T12:48:44.479323+0000","last_unstale":"2024-04-28T12:49:35.631237+0000","last_undegraded":"2024-04-28T12:49:35.631237+0000","last_fullsized":"2024-04-28T12:49:35.631237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:40:47.274081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.633795+0000","last_change":"2024-04-28T12:48:44.472305+0000","last_active":"2024-04-28T12:49:35.633795+0000","last_peered":"2024-04-28T12:49:35.633795+0000","last_clean":"2024-04-28T12:49:35.633795+0000","last_became_active":"2024-04-28T12:48:44.471227+0000","last_became_peered":"2024-04-28T12:48:44.471227+0000","last_unstale":"2024-04-28T12:49:35.633795+0000","last_undegraded":"2024-04-28T12:49:35.633795+0000","last_fullsized":"2024-04-28T12:49:35.633795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:27:14.602075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.811084+0000","last_change":"2024-04-28T12:48:44.490866+0000","last_active":"2024-04-28T12:49:35.811084+0000","last_peered":"2024-04-28T12:49:35.811084+0000","last_clean":"2024-04-28T12:49:35.811084+0000","last_became_active":"2024-04-28T12:48:44.490515+0000","last_became_peered":"2024-04-28T12:48:44.490515+0000","last_unstale":"2024-04-28T12:49:35.811084+0000","last_undegraded":"2024-04-28T12:49:35.811084+0000","last_fullsized":"2024-04-28T12:49:35.811084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:19:22.404853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074784+0000","last_change":"2024-04-28T12:48:44.480611+0000","last_active":"2024-04-28T12:49:36.074784+0000","last_peered":"2024-04-28T12:49:36.074784+0000","last_clean":"2024-04-28T12:49:36.074784+0000","last_became_active":"2024-04-28T12:48:44.480294+0000","last_became_peered":"2024-04-28T12:48:44.480294+0000","last_unstale":"2024-04-28T12:49:36.074784+0000","last_undegraded":"2024-04-28T12:49:36.074784+0000","last_fullsized":"2024-04-28T12:49:36.074784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:55.450825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.810781+0000","last_change":"2024-04-28T12:48:44.483003+0000","last_active":"2024-04-28T12:49:35.810781+0000","last_peered":"2024-04-28T12:49:35.810781+0000","last_clean":"2024-04-28T12:49:35.810781+0000","last_became_active":"2024-04-28T12:48:44.482630+0000","last_became_peered":"2024-04-28T12:48:44.482630+0000","last_unstale":"2024-04-28T12:49:35.810781+0000","last_undegraded":"2024-04-28T12:49:35.810781+0000","last_fullsized":"2024-04-28T12:49:35.810781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:05:58.358655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.638350+0000","last_change":"2024-04-28T12:48:44.489780+0000","last_active":"2024-04-28T12:49:35.638350+0000","last_peered":"2024-04-28T12:49:35.638350+0000","last_clean":"2024-04-28T12:49:35.638350+0000","last_became_active":"2024-04-28T12:48:44.489276+0000","last_became_peered":"2024-04-28T12:48:44.489276+0000","last_unstale":"2024-04-28T12:49:35.638350+0000","last_undegraded":"2024-04-28T12:49:35.638350+0000","last_fullsized":"2024-04-28T12:49:35.638350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:49.264624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:36.074471+0000","last_change":"2024-04-28T12:48:44.472196+0000","last_active":"2024-04-28T12:49:36.074471+0000","last_peered":"2024-04-28T12:49:36.074471+0000","last_clean":"2024-04-28T12:49:36.074471+0000","last_became_active":"2024-04-28T12:48:44.471857+0000","last_became_peered":"2024-04-28T12:48:44.471857+0000","last_unstale":"2024-04-28T12:49:36.074471+0000","last_undegraded":"2024-04-28T12:49:36.074471+0000","last_fullsized":"2024-04-28T12:49:36.074471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:43.813225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-28T12:49:35.630551+0000","last_change":"2024-04-28T12:48:44.478788+0000","last_active":"2024-04-28T12:49:35.630551+0000","last_peered":"2024-04-28T12:49:35.630551+0000","last_clean":"2024-04-28T12:49:35.630551+0000","last_became_active":"2024-04-28T12:48:44.478449+0000","last_became_peered":"2024-04-28T12:48:44.478449+0000","last_unstale":"2024-04-28T12:49:35.630551+0000","last_undegraded":"2024-04-28T12:49:35.630551+0000","last_fullsized":"2024-04-28T12:49:35.630551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:43.491114+0000","last_clean_scrub_stamp":"2024-04-28T12:48:43.491114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:48:11.638628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283884,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54008,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":438,"num_read_kb":276,"num_write":152,"num_write_kb":95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":50420,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":162024,"data_compressed":11972,"data_compressed_allocated":20480,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":459357,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1101214,"data_compressed":43734,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1598,"internal_metadata":31914434},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":966656,"data_stored":459571,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1598,"internal_metadata":36174274},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037806,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":459469,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299443,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1051030,"data_compressed":41890,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1587,"internal_metadata":31914445},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":458960,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":509151,"data_compressed":7635,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":460231,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460103,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":459271,"data_compressed":5791,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888130,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":509795,"data_compressed":14075,"data_compressed_allocated":20480,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":1050714,"data_compressed":41890,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34463,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36099,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":50398,"data_compressed":1844,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":51042,"data_compressed":8284,"data_compressed_allocated":12288,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":51976,"data_compressed":1844,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:50:04.326 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-28T12:50:04.326 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-28T12:50:04.327 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-28T12:50:04.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-28T12:50:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:04.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:04 smithi077 ceph-mon[32553]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:05.004 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:04 smithi053 ceph-mon[29182]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:04 smithi191 ceph-mon[33016]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:05.945 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:05 smithi053 ceph-mon[29182]: pgmap v333: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:05.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:05 smithi077 ceph-mon[32553]: pgmap v333: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:06.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:05 smithi191 ceph-mon[33016]: pgmap v333: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:06.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 -- 172.21.15.53:0/1202804728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 msgr2=0x7fc990109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 --2- 172.21.15.53:0/1202804728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc990109940 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc9800099c0 tx=0x7fc980009cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 -- 172.21.15.53:0/1202804728 shutdown_connections 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 --2- 172.21.15.53:0/1202804728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc990109940 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 --2- 172.21.15.53:0/1202804728 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc990103e80 0x7fc990110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 --2- 172.21.15.53:0/1202804728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9901034c0 0x7fc990103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 -- 172.21.15.53:0/1202804728 >> 172.21.15.53:0/1202804728 conn(0x7fc99007a5d0 msgr2=0x7fc99007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 -- 172.21.15.53:0/1202804728 shutdown_connections 2024-04-28T12:50:06.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 -- 172.21.15.53:0/1202804728 wait complete. 2024-04-28T12:50:06.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 Processor -- start 2024-04-28T12:50:06.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.256+0000 7fc99632c700 1 -- start start 2024-04-28T12:50:06.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc99632c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9901034c0 0x7fc990075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:06.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc99632c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc990103e80 0x7fc990075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:06.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc99632c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc9900761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:06.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc99632c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc990113640 con 0x7fc990109560 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc99632c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc9901134c0 con 0x7fc9901034c0 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc99632c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc9901137c0 con 0x7fc990103e80 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc9900761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc9900761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43720/0 (socket says 172.21.15.53:43720) 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 -- 172.21.15.53:0/3707610676 learned_addr learned my addr 172.21.15.53:0/3707610676 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc98f7fe700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc990103e80 0x7fc990075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc98ffff700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9901034c0 0x7fc990075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 -- 172.21.15.53:0/3707610676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc990103e80 msgr2=0x7fc990075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc990103e80 0x7fc990075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 -- 172.21.15.53:0/3707610676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9901034c0 msgr2=0x7fc990075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9901034c0 0x7fc990075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 -- 172.21.15.53:0/3707610676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9901a7f80 con 0x7fc990109560 2024-04-28T12:50:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.257+0000 7fc9948c9700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc9900761a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc98800ada0 tx=0x7fc98800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.258+0000 7fc98d7fa700 1 -- 172.21.15.53:0/3707610676 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988017020 con 0x7fc990109560 2024-04-28T12:50:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.258+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9901a8270 con 0x7fc990109560 2024-04-28T12:50:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.258+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9901a87b0 con 0x7fc990109560 2024-04-28T12:50:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.258+0000 7fc98d7fa700 1 -- 172.21.15.53:0/3707610676 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc98800cd80 con 0x7fc990109560 2024-04-28T12:50:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.258+0000 7fc98d7fa700 1 -- 172.21.15.53:0/3707610676 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988005260 con 0x7fc990109560 2024-04-28T12:50:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.259+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc99010b9f0 con 0x7fc990109560 2024-04-28T12:50:06.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.260+0000 7fc98d7fa700 1 -- 172.21.15.53:0/3707610676 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc98800a090 con 0x7fc990109560 2024-04-28T12:50:06.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.260+0000 7fc98d7fa700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc9780a3d50 0x7fc9780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:06.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.261+0000 7fc98ffff700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc9780a3d50 0x7fc9780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:06.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.261+0000 7fc98d7fa700 1 -- 172.21.15.53:0/3707610676 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fc9880d01a0 con 0x7fc990109560 2024-04-28T12:50:06.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.261+0000 7fc98ffff700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc9780a3d50 0x7fc9780a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc980009990 tx=0x7fc980009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:06.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.264+0000 7fc98d7fa700 1 -- 172.21.15.53:0/3707610676 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc988021020 con 0x7fc990109560 2024-04-28T12:50:06.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.526+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc990076c50 con 0x7fc990109560 2024-04-28T12:50:06.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.526+0000 7fc98d7fa700 1 -- 172.21.15.53:0/3707610676 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc98809cab0 con 0x7fc990109560 2024-04-28T12:50:06.523 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:06.523 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-28T12:50:06.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc9780a3d50 msgr2=0x7fc9780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:06.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc9780a3d50 0x7fc9780a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc980009990 tx=0x7fc980009650 comp rx=0 tx=0).stop 2024-04-28T12:50:06.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 msgr2=0x7fc9900761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:06.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc9900761a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc98800ada0 tx=0x7fc98800c330 comp rx=0 tx=0).stop 2024-04-28T12:50:06.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 shutdown_connections 2024-04-28T12:50:06.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc9780a3d50 0x7fc9780a6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc990109560 0x7fc9900761a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9901034c0 0x7fc990075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 --2- 172.21.15.53:0/3707610676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc990103e80 0x7fc990075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:06.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 >> 172.21.15.53:0/3707610676 conn(0x7fc99007a5d0 msgr2=0x7fc990102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:06.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 shutdown_connections 2024-04-28T12:50:06.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:06.529+0000 7fc99632c700 1 -- 172.21.15.53:0/3707610676 wait complete. 2024-04-28T12:50:06.850 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:06 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3707610676' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T12:50:06.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:06 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3707610676' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T12:50:07.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:06 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3707610676' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T12:50:07.162 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-28T12:50:07.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:07.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 -- 172.21.15.53:0/4290752844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4104550 msgr2=0x7f72f410ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 --2- 172.21.15.53:0/4290752844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4104550 0x7f72f410ade0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009ca0 tx=0x7f72ec02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 -- 172.21.15.53:0/4290752844 shutdown_connections 2024-04-28T12:50:07.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 --2- 172.21.15.53:0/4290752844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4104550 0x7f72f410ade0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 --2- 172.21.15.53:0/4290752844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72f4103b90 0x7f72f4104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 --2- 172.21.15.53:0/4290752844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f41029a0 0x7f72f4102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 -- 172.21.15.53:0/4290752844 >> 172.21.15.53:0/4290752844 conn(0x7f72f40fdf30 msgr2=0x7f72f4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 -- 172.21.15.53:0/4290752844 shutdown_connections 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.272+0000 7f72fae47640 1 -- 172.21.15.53:0/4290752844 wait complete. 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 Processor -- start 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 -- start start 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f41029a0 0x7f72f40766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4103b90 0x7f72f4074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72f4104550 0x7f72f4075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f72f410e4b0 con 0x7f72f4103b90 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72f410e330 con 0x7f72f41029a0 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72fae47640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f72f410e630 con 0x7f72f4104550 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4103b90 0x7f72f4074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4103b90 0x7f72f4074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35884/0 (socket says 172.21.15.53:35884) 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 -- 172.21.15.53:0/3947540096 learned_addr learned my addr 172.21.15.53:0/3947540096 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f8be3640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f41029a0 0x7f72f40766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f93e4640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72f4104550 0x7f72f4075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 -- 172.21.15.53:0/3947540096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72f4104550 msgr2=0x7f72f4075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72f4104550 0x7f72f4075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 -- 172.21.15.53:0/3947540096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f41029a0 msgr2=0x7f72f40766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f41029a0 0x7f72f40766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f3fff640 1 -- 172.21.15.53:0/3947540096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f4075af0 con 0x7f72f4103b90 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.273+0000 7f72f93e4640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72f4104550 0x7f72f4075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.274+0000 7f72f3fff640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4103b90 0x7f72f4074d20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f72e800d950 tx=0x7f72e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.274+0000 7f72f1ffb640 1 -- 172.21.15.53:0/3947540096 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8017070 con 0x7f72f4103b90 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.274+0000 7f72f1ffb640 1 -- 172.21.15.53:0/3947540096 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72e8009c30 con 0x7f72f4103b90 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.274+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f41a2c80 con 0x7f72f4103b90 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.274+0000 7f72f1ffb640 1 -- 172.21.15.53:0/3947540096 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e800f620 con 0x7f72f4103b90 2024-04-28T12:50:07.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.274+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f41a31d0 con 0x7f72f4103b90 2024-04-28T12:50:07.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.275+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72f4109cc0 con 0x7f72f4103b90 2024-04-28T12:50:07.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.276+0000 7f72f1ffb640 1 -- 172.21.15.53:0/3947540096 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f72e8010460 con 0x7f72f4103b90 2024-04-28T12:50:07.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.276+0000 7f72f1ffb640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f72dc0a3c40 0x7f72dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.277+0000 7f72f8be3640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f72dc0a3c40 0x7f72dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.277+0000 7f72f1ffb640 1 -- 172.21.15.53:0/3947540096 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f72e80d0220 con 0x7f72f4103b90 2024-04-28T12:50:07.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.277+0000 7f72f8be3640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f72dc0a3c40 0x7f72dc0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f72f41039f0 tx=0x7f72e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:07.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.281+0000 7f72f1ffb640 1 -- 172.21.15.53:0/3947540096 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f72e809ca60 con 0x7f72f4103b90 2024-04-28T12:50:07.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.468+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72f402dc50 con 0x7f72f4103b90 2024-04-28T12:50:07.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.470+0000 7f72f1ffb640 1 -- 172.21.15.53:0/3947540096 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7f72e8009de0 con 0x7f72f4103b90 2024-04-28T12:50:07.467 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:07.467 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:07.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f72dc0a3c40 msgr2=0x7f72dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f72dc0a3c40 0x7f72dc0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f72f41039f0 tx=0x7f72e4005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:07.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4103b90 msgr2=0x7f72f4074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4103b90 0x7f72f4074d20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f72e800d950 tx=0x7f72e800de10 comp rx=0 tx=0).stop 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 shutdown_connections 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f72dc0a3c40 0x7f72dc0a6100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72f4104550 0x7f72f4075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72f4103b90 0x7f72f4074d20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 --2- 172.21.15.53:0/3947540096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f41029a0 0x7f72f40766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 >> 172.21.15.53:0/3947540096 conn(0x7f72f40fdf30 msgr2=0x7f72f4106e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 shutdown_connections 2024-04-28T12:50:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.472+0000 7f72fae47640 1 -- 172.21.15.53:0/3947540096 wait complete. 2024-04-28T12:50:07.480 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:42.284772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:42.832857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:50:07.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 -- 172.21.15.53:0/2025244793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 msgr2=0x7f05a010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 --2- 172.21.15.53:0/2025244793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a010add0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0598009ca0 tx=0x7f059802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 -- 172.21.15.53:0/2025244793 shutdown_connections 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 --2- 172.21.15.53:0/2025244793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a010add0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 --2- 172.21.15.53:0/2025244793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a0103b80 0x7f05a0104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 --2- 172.21.15.53:0/2025244793 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a0102990 0x7f05a0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 -- 172.21.15.53:0/2025244793 >> 172.21.15.53:0/2025244793 conn(0x7f05a00fdf00 msgr2=0x7f05a0100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 -- 172.21.15.53:0/2025244793 shutdown_connections 2024-04-28T12:50:07.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.589+0000 7f05a4b89640 1 -- 172.21.15.53:0/2025244793 wait complete. 2024-04-28T12:50:07.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 Processor -- start 2024-04-28T12:50:07.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 -- start start 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a0102990 0x7f05a0198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a0103b80 0x7f05a0199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a01a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f05a010e4a0 con 0x7f05a0104540 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05a010e320 con 0x7f05a0103b80 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f05a4b89640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f05a010e620 con 0x7f05a0102990 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a01a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a01a0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35898/0 (socket says 172.21.15.53:35898) 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 -- 172.21.15.53:0/1776758676 learned_addr learned my addr 172.21.15.53:0/1776758676 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059e59c640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a0102990 0x7f05a0198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059dd9b640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a0103b80 0x7f05a0199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 -- 172.21.15.53:0/1776758676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a0102990 msgr2=0x7f05a0198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a0102990 0x7f05a0198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 -- 172.21.15.53:0/1776758676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a0103b80 msgr2=0x7f05a0199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a0103b80 0x7f05a0199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059ed9d640 1 -- 172.21.15.53:0/1776758676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a01a0700 con 0x7f05a0104540 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.590+0000 7f059dd9b640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a0103b80 0x7f05a0199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.591+0000 7f059ed9d640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a01a0090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0598031a20 tx=0x7f0598003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.591+0000 7f058f7fe640 1 -- 172.21.15.53:0/1776758676 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598004240 con 0x7f05a0104540 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.591+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a01a0990 con 0x7f05a0104540 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.591+0000 7f058f7fe640 1 -- 172.21.15.53:0/1776758676 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0598003e10 con 0x7f05a0104540 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.591+0000 7f058f7fe640 1 -- 172.21.15.53:0/1776758676 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980305c0 con 0x7f05a0104540 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.591+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a01a0ee0 con 0x7f05a0104540 2024-04-28T12:50:07.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.592+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a0109cb0 con 0x7f05a0104540 2024-04-28T12:50:07.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.593+0000 7f058f7fe640 1 -- 172.21.15.53:0/1776758676 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f059802f660 con 0x7f05a0104540 2024-04-28T12:50:07.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.593+0000 7f058f7fe640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3c40 0x7f05880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.594+0000 7f059e59c640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3c40 0x7f05880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.594+0000 7f058f7fe640 1 -- 172.21.15.53:0/1776758676 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f05980f1d50 con 0x7f05a0104540 2024-04-28T12:50:07.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.594+0000 7f059e59c640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3c40 0x7f05880a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f05a01039e0 tx=0x7f0590005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:07.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.598+0000 7f058f7fe640 1 -- 172.21.15.53:0/1776758676 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f05980be6e0 con 0x7f05a0104540 2024-04-28T12:50:07.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.785+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f05a0063530 con 0x7f05a0104540 2024-04-28T12:50:07.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.786+0000 7f058f7fe640 1 -- 172.21.15.53:0/1776758676 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f05980bdfe0 con 0x7f05a0104540 2024-04-28T12:50:07.782 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T12:50:07.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3c40 msgr2=0x7f05880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3c40 0x7f05880a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f05a01039e0 tx=0x7f0590005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 msgr2=0x7f05a01a0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a01a0090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0598031a20 tx=0x7f0598003c20 comp rx=0 tx=0).stop 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 shutdown_connections 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f05880a3c40 0x7f05880a6100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05a0104540 0x7f05a01a0090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a0103b80 0x7f05a0199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 --2- 172.21.15.53:0/1776758676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05a0102990 0x7f05a0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 >> 172.21.15.53:0/1776758676 conn(0x7f05a00fdf00 msgr2=0x7f05a0106e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 shutdown_connections 2024-04-28T12:50:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.788+0000 7f05a4b89640 1 -- 172.21.15.53:0/1776758676 wait complete. 2024-04-28T12:50:07.794 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:07 smithi053 ceph-mon[29182]: pgmap v334: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:50:07.794 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:07 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3947540096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:07.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.903+0000 7f49c81b4640 1 -- 172.21.15.53:0/1903868716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0101870 msgr2=0x7f49c0101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.903+0000 7f49c81b4640 1 --2- 172.21.15.53:0/1903868716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0101870 0x7f49c0101c90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009900 tx=0x7f49bc02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.903+0000 7f49c81b4640 1 -- 172.21.15.53:0/1903868716 shutdown_connections 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 --2- 172.21.15.53:0/1903868716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49c0103420 0x7f49c0109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 --2- 172.21.15.53:0/1903868716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49c0102a60 0x7f49c0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 --2- 172.21.15.53:0/1903868716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0101870 0x7f49c0101c90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 -- 172.21.15.53:0/1903868716 >> 172.21.15.53:0/1903868716 conn(0x7f49c00fcde0 msgr2=0x7f49c00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 -- 172.21.15.53:0/1903868716 shutdown_connections 2024-04-28T12:50:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 -- 172.21.15.53:0/1903868716 wait complete. 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 Processor -- start 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 -- start start 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49c0101870 0x7f49c0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49c0102a60 0x7f49c0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0103420 0x7f49c019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f49c010d390 con 0x7f49c0103420 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49c010d210 con 0x7f49c0101870 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c81b4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f49c010d510 con 0x7f49c0102a60 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c6751640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0103420 0x7f49c019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c6751640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0103420 0x7f49c019f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35918/0 (socket says 172.21.15.53:35918) 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.904+0000 7f49c6751640 1 -- 172.21.15.53:0/4277249920 learned_addr learned my addr 172.21.15.53:0/4277249920 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c5f50640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49c0101870 0x7f49c0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c574f640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49c0102a60 0x7f49c0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c6751640 1 -- 172.21.15.53:0/4277249920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49c0102a60 msgr2=0x7f49c0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c6751640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49c0102a60 0x7f49c0198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c6751640 1 -- 172.21.15.53:0/4277249920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49c0101870 msgr2=0x7f49c0197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c6751640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49c0101870 0x7f49c0197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c6751640 1 -- 172.21.15.53:0/4277249920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c019f760 con 0x7f49c0103420 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c6751640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0103420 0x7f49c019f0f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009fd0 tx=0x7f49b800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49b6ffd640 1 -- 172.21.15.53:0/4277249920 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8017070 con 0x7f49c0103420 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49b6ffd640 1 -- 172.21.15.53:0/4277249920 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b8003e20 con 0x7f49c0103420 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c019fa50 con 0x7f49c0103420 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49b6ffd640 1 -- 172.21.15.53:0/4277249920 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8010460 con 0x7f49c0103420 2024-04-28T12:50:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.905+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c019ff50 con 0x7f49c0103420 2024-04-28T12:50:07.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.906+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c0108d50 con 0x7f49c0103420 2024-04-28T12:50:07.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.907+0000 7f49b6ffd640 1 -- 172.21.15.53:0/4277249920 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f49b8003970 con 0x7f49c0103420 2024-04-28T12:50:07.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.908+0000 7f49b6ffd640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f49ac0a3c40 0x7f49ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:07.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.908+0000 7f49c5f50640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f49ac0a3c40 0x7f49ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:07.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.908+0000 7f49b6ffd640 1 -- 172.21.15.53:0/4277249920 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f49b80d02c0 con 0x7f49c0103420 2024-04-28T12:50:07.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.908+0000 7f49c5f50640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f49ac0a3c40 0x7f49ac0a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009230 tx=0x7f49bc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:07.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:07.912+0000 7f49b6ffd640 1 -- 172.21.15.53:0/4277249920 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f49b809cb40 con 0x7f49c0103420 2024-04-28T12:50:07.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:07 smithi077 ceph-mon[32553]: pgmap v334: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:50:07.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:07 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3947540096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:07 smithi191 ceph-mon[33016]: pgmap v334: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:50:08.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:07 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3947540096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:08.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.101+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f49c002dd20 con 0x7f49c0103420 2024-04-28T12:50:08.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.101+0000 7f49b6ffd640 1 -- 172.21.15.53:0/4277249920 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f49b809c440 con 0x7f49c0103420 2024-04-28T12:50:08.097 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f49ac0a3c40 msgr2=0x7f49ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f49ac0a3c40 0x7f49ac0a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009230 tx=0x7f49bc0314a0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0103420 msgr2=0x7f49c019f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0103420 0x7f49c019f0f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009fd0 tx=0x7f49b800b260 comp rx=0 tx=0).stop 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 shutdown_connections 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f49ac0a3c40 0x7f49ac0a6100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0103420 0x7f49c019f0f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49c0102a60 0x7f49c0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 --2- 172.21.15.53:0/4277249920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49c0101870 0x7f49c0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 >> 172.21.15.53:0/4277249920 conn(0x7f49c00fcde0 msgr2=0x7f49c0105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 shutdown_connections 2024-04-28T12:50:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.103+0000 7f49c81b4640 1 -- 172.21.15.53:0/4277249920 wait complete. 2024-04-28T12:50:08.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:50:08.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 -- 172.21.15.53:0/2525147640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 msgr2=0x7fe408101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2525147640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408101c80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe404009900 tx=0x7fe40402a410 comp rx=0 tx=0).stop 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 -- 172.21.15.53:0/2525147640 shutdown_connections 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2525147640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe408103410 0x7fe408109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2525147640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe408102a50 0x7fe408102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2525147640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408101c80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 -- 172.21.15.53:0/2525147640 >> 172.21.15.53:0/2525147640 conn(0x7fe4080fcdd0 msgr2=0x7fe4080ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 -- 172.21.15.53:0/2525147640 shutdown_connections 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.219+0000 7fe40fec5640 1 -- 172.21.15.53:0/2525147640 wait complete. 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 Processor -- start 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 -- start start 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe408102a50 0x7fe408199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe408103410 0x7fe4081a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe40810e470 con 0x7fe408101860 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe40810e2f0 con 0x7fe408103410 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40fec5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe40810e5f0 con 0x7fe408102a50 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35932/0 (socket says 172.21.15.53:35932) 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 -- 172.21.15.53:0/2244042905 learned_addr learned my addr 172.21.15.53:0/2244042905 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40e462640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe408103410 0x7fe4081a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40d460640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe408102a50 0x7fe408199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 -- 172.21.15.53:0/2244042905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe408102a50 msgr2=0x7fe408199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe408102a50 0x7fe408199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 -- 172.21.15.53:0/2244042905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe408103410 msgr2=0x7fe4081a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe408103410 0x7fe4081a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40dc61640 1 -- 172.21.15.53:0/2244042905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4081a07f0 con 0x7fe408101860 2024-04-28T12:50:08.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.220+0000 7fe40d460640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe408102a50 0x7fe408199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:08.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.221+0000 7fe40dc61640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408198bd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe404006010 tx=0x7fe4040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:08.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.221+0000 7fe3feffd640 1 -- 172.21.15.53:0/2244042905 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40403f030 con 0x7fe408101860 2024-04-28T12:50:08.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.221+0000 7fe3feffd640 1 -- 172.21.15.53:0/2244042905 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe404003cb0 con 0x7fe408101860 2024-04-28T12:50:08.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.221+0000 7fe3feffd640 1 -- 172.21.15.53:0/2244042905 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe404035070 con 0x7fe408101860 2024-04-28T12:50:08.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.221+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4081a0a80 con 0x7fe408101860 2024-04-28T12:50:08.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.221+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4081a0fc0 con 0x7fe408101860 2024-04-28T12:50:08.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.222+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe408108cf0 con 0x7fe408101860 2024-04-28T12:50:08.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.223+0000 7fe3feffd640 1 -- 172.21.15.53:0/2244042905 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe404042030 con 0x7fe408101860 2024-04-28T12:50:08.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.224+0000 7fe3feffd640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe3f40a3cc0 0x7fe3f40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.224+0000 7fe40d460640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe3f40a3cc0 0x7fe3f40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.224+0000 7fe3feffd640 1 -- 172.21.15.53:0/2244042905 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fe4040f0bb0 con 0x7fe408101860 2024-04-28T12:50:08.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.224+0000 7fe40d460640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe3f40a3cc0 0x7fe3f40a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80058c0 tx=0x7fe3f8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:08.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.228+0000 7fe3feffd640 1 -- 172.21.15.53:0/2244042905 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe4040bd3f0 con 0x7fe408101860 2024-04-28T12:50:08.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.414+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe408063530 con 0x7fe408101860 2024-04-28T12:50:08.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.415+0000 7fe3feffd640 1 -- 172.21.15.53:0/2244042905 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fe4040bccf0 con 0x7fe408101860 2024-04-28T12:50:08.411 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe3f40a3cc0 msgr2=0x7fe3f40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe3f40a3cc0 0x7fe3f40a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80058c0 tx=0x7fe3f8005850 comp rx=0 tx=0).stop 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 msgr2=0x7fe408198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408198bd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe404006010 tx=0x7fe4040042f0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 shutdown_connections 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fe3f40a3cc0 0x7fe3f40a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe408103410 0x7fe4081a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe408102a50 0x7fe408199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 --2- 172.21.15.53:0/2244042905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe408101860 0x7fe408198bd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 >> 172.21.15.53:0/2244042905 conn(0x7fe4080fcdd0 msgr2=0x7fe408105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 shutdown_connections 2024-04-28T12:50:08.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.417+0000 7fe40fec5640 1 -- 172.21.15.53:0/2244042905 wait complete. 2024-04-28T12:50:08.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:08.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 -- 172.21.15.53:0/1577059608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 msgr2=0x7f868c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 --2- 172.21.15.53:0/1577059608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c102ee0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f867c009900 tx=0x7f867c02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:08.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 -- 172.21.15.53:0/1577059608 shutdown_connections 2024-04-28T12:50:08.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 --2- 172.21.15.53:0/1577059608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c103420 0x7f868c109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 --2- 172.21.15.53:0/1577059608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c102ee0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 --2- 172.21.15.53:0/1577059608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c101870 0x7f868c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 -- 172.21.15.53:0/1577059608 >> 172.21.15.53:0/1577059608 conn(0x7f868c0fcde0 msgr2=0x7f868c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 -- 172.21.15.53:0/1577059608 shutdown_connections 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.531+0000 7f86946b8640 1 -- 172.21.15.53:0/1577059608 wait complete. 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 Processor -- start 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 -- start start 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c101870 0x7f868c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c1991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c103420 0x7f868c1a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f868c10d460 con 0x7f868c102a60 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f868c10d2e0 con 0x7f868c103420 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f86946b8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f868c10d5e0 con 0x7f868c101870 2024-04-28T12:50:08.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c1991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c1991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35948/0 (socket says 172.21.15.53:35948) 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 -- 172.21.15.53:0/560076947 learned_addr learned my addr 172.21.15.53:0/560076947 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8692454640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c101870 0x7f868c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8692c55640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c103420 0x7f868c1a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 -- 172.21.15.53:0/560076947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c101870 msgr2=0x7f868c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c101870 0x7f868c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 -- 172.21.15.53:0/560076947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c103420 msgr2=0x7f868c1a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c103420 0x7f868c1a0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 -- 172.21.15.53:0/560076947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c1a08d0 con 0x7f868c102a60 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8692c55640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c103420 0x7f868c1a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.532+0000 7f8691c53640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c1991d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f867c005ab0 tx=0x7f867c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.533+0000 7f86837fe640 1 -- 172.21.15.53:0/560076947 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c03f070 con 0x7f868c102a60 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.533+0000 7f86837fe640 1 -- 172.21.15.53:0/560076947 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f867c03b460 con 0x7f868c102a60 2024-04-28T12:50:08.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.533+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868c1a0b60 con 0x7f868c102a60 2024-04-28T12:50:08.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.533+0000 7f86837fe640 1 -- 172.21.15.53:0/560076947 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c035030 con 0x7f868c102a60 2024-04-28T12:50:08.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.533+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f868c1a1050 con 0x7f868c102a60 2024-04-28T12:50:08.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.534+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f868c1a1280 con 0x7f868c102a60 2024-04-28T12:50:08.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.535+0000 7f86837fe640 1 -- 172.21.15.53:0/560076947 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f867c042070 con 0x7f868c102a60 2024-04-28T12:50:08.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.536+0000 7f86837fe640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86780a3c40 0x7f86780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.536+0000 7f8692454640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86780a3c40 0x7f86780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.536+0000 7f86837fe640 1 -- 172.21.15.53:0/560076947 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f867c0f1360 con 0x7f868c102a60 2024-04-28T12:50:08.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.536+0000 7f8692454640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86780a3c40 0x7f86780a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f868c199b60 tx=0x7f8688008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:08.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.539+0000 7f86837fe640 1 -- 172.21.15.53:0/560076947 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f867c0bdba0 con 0x7f868c102a60 2024-04-28T12:50:08.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.768+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f868c1a1570 con 0x7f868c102a60 2024-04-28T12:50:08.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.768+0000 7f86837fe640 1 -- 172.21.15.53:0/560076947 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11873 (secure 0 0 0) 0x7f867c0bd4a0 con 0x7f868c102a60 2024-04-28T12:50:08.765 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:08.765 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:48:49.455677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24485,24478]},"id":1}]} 2024-04-28T12:50:08.765 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 22 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86780a3c40 msgr2=0x7f86780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86780a3c40 0x7f86780a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f868c199b60 tx=0x7f8688008040 comp rx=0 tx=0).stop 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 msgr2=0x7f868c1991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c1991d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f867c005ab0 tx=0x7f867c003cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 shutdown_connections 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f86780a3c40 0x7f86780a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c103420 0x7f868c1a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f868c102a60 0x7f868c1991d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 --2- 172.21.15.53:0/560076947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c101870 0x7f868c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 >> 172.21.15.53:0/560076947 conn(0x7f868c0fcde0 msgr2=0x7f868c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:08.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.770+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 shutdown_connections 2024-04-28T12:50:08.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.771+0000 7f86946b8640 1 -- 172.21.15.53:0/560076947 wait complete. 2024-04-28T12:50:08.778 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:08.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 -- 172.21.15.53:0/2171006114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90102a80 msgr2=0x7f2c90102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 --2- 172.21.15.53:0/2171006114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90102a80 0x7f2c90102f00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009900 tx=0x7f2c7802a410 comp rx=0 tx=0).stop 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 -- 172.21.15.53:0/2171006114 shutdown_connections 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 --2- 172.21.15.53:0/2171006114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c90103440 0x7f2c90109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 --2- 172.21.15.53:0/2171006114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90102a80 0x7f2c90102f00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 --2- 172.21.15.53:0/2171006114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90101890 0x7f2c90101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 -- 172.21.15.53:0/2171006114 >> 172.21.15.53:0/2171006114 conn(0x7f2c900fce20 msgr2=0x7f2c900ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.884+0000 7f2c95847640 1 -- 172.21.15.53:0/2171006114 shutdown_connections 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 -- 172.21.15.53:0/2171006114 wait complete. 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 Processor -- start 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 -- start start 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c90101890 0x7f2c9019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90102a80 0x7f2c9019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90103440 0x7f2c901a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9010d3c0 con 0x7f2c90103440 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9010d240 con 0x7f2c90102a80 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c95847640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9010d540 con 0x7f2c90101890 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90103440 0x7f2c901a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90103440 0x7f2c901a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35962/0 (socket says 172.21.15.53:35962) 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.885+0000 7f2c8f7fe640 1 -- 172.21.15.53:0/1600189698 learned_addr learned my addr 172.21.15.53:0/1600189698 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c8effd640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c90101890 0x7f2c9019ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c87fff640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90102a80 0x7f2c9019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c8f7fe640 1 -- 172.21.15.53:0/1600189698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c90101890 msgr2=0x7f2c9019ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c8f7fe640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c90101890 0x7f2c9019ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c8f7fe640 1 -- 172.21.15.53:0/1600189698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90102a80 msgr2=0x7f2c9019b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c8f7fe640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90102a80 0x7f2c9019b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c8f7fe640 1 -- 172.21.15.53:0/1600189698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c901a2a30 con 0x7f2c90103440 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c87fff640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90102a80 0x7f2c9019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:08.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c8f7fe640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90103440 0x7f2c901a23c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800cad0 tx=0x7f2c8800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:08.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c94845640 1 -- 172.21.15.53:0/1600189698 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8800ae20 con 0x7f2c90103440 2024-04-28T12:50:08.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c94845640 1 -- 172.21.15.53:0/1600189698 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c88005020 con 0x7f2c90103440 2024-04-28T12:50:08.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c94845640 1 -- 172.21.15.53:0/1600189698 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8800f6b0 con 0x7f2c90103440 2024-04-28T12:50:08.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c901a2d20 con 0x7f2c90103440 2024-04-28T12:50:08.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.886+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c901a3210 con 0x7f2c90103440 2024-04-28T12:50:08.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.887+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c901a34a0 con 0x7f2c90103440 2024-04-28T12:50:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.888+0000 7f2c94845640 1 -- 172.21.15.53:0/1600189698 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2c880051d0 con 0x7f2c90103440 2024-04-28T12:50:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.889+0000 7f2c94845640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c7c0a3c40 0x7f2c7c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.889+0000 7f2c8effd640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c7c0a3c40 0x7f2c7c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.889+0000 7f2c94845640 1 -- 172.21.15.53:0/1600189698 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f2c880d0670 con 0x7f2c90103440 2024-04-28T12:50:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.889+0000 7f2c8effd640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c7c0a3c40 0x7f2c7c0a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2c80005fd0 tx=0x7f2c80005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:08.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:08.893+0000 7f2c94845640 1 -- 172.21.15.53:0/1600189698 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2c88014030 con 0x7f2c90103440 2024-04-28T12:50:08.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:08 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1776758676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:08.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:08 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4277249920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:08.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:08 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2244042905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:08 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1776758676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:08 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4277249920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:08 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2244042905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:09.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.079+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c901a3800 con 0x7f2c90103440 2024-04-28T12:50:09.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.081+0000 7f2c94845640 1 -- 172.21.15.53:0/1600189698 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7f2c8809cef0 con 0x7f2c90103440 2024-04-28T12:50:09.077 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:09.078 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:09.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c7c0a3c40 msgr2=0x7f2c7c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c7c0a3c40 0x7f2c7c0a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2c80005fd0 tx=0x7f2c80005850 comp rx=0 tx=0).stop 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90103440 msgr2=0x7f2c901a23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90103440 0x7f2c901a23c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800cad0 tx=0x7f2c8800cf90 comp rx=0 tx=0).stop 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 shutdown_connections 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f2c7c0a3c40 0x7f2c7c0a6100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c90103440 0x7f2c901a23c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90102a80 0x7f2c9019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 --2- 172.21.15.53:0/1600189698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c90101890 0x7f2c9019ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 >> 172.21.15.53:0/1600189698 conn(0x7f2c900fce20 msgr2=0x7f2c900fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 shutdown_connections 2024-04-28T12:50:09.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.083+0000 7f2c95847640 1 -- 172.21.15.53:0/1600189698 wait complete. 2024-04-28T12:50:09.089 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:08 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1776758676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:09.090 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:08 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4277249920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:09.090 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:08 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2244042905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:09.091 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:42.284772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:42.832857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:50:09.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:50:09.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 -- 172.21.15.53:0/262037640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30103440 msgr2=0x7faa30109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 --2- 172.21.15.53:0/262037640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30103440 0x7faa30109cd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faa28009ca0 tx=0x7faa2802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 -- 172.21.15.53:0/262037640 shutdown_connections 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 --2- 172.21.15.53:0/262037640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30103440 0x7faa30109cd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 --2- 172.21.15.53:0/262037640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa30102a80 0x7faa30102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 --2- 172.21.15.53:0/262037640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa30101890 0x7faa30101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 -- 172.21.15.53:0/262037640 >> 172.21.15.53:0/262037640 conn(0x7faa300fce20 msgr2=0x7faa300ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 -- 172.21.15.53:0/262037640 shutdown_connections 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 -- 172.21.15.53:0/262037640 wait complete. 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 Processor -- start 2024-04-28T12:50:09.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.200+0000 7faa36978640 1 -- start start 2024-04-28T12:50:09.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa36978640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30101890 0x7faa30198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa36978640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa30102a80 0x7faa30199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa36978640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa30103440 0x7faa301a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa36978640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d3c0 con 0x7faa30101890 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa36978640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d240 con 0x7faa30103440 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa36978640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d540 con 0x7faa30102a80 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30101890 0x7faa30198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30101890 0x7faa30198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35978/0 (socket says 172.21.15.53:35978) 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 -- 172.21.15.53:0/1811129523 learned_addr learned my addr 172.21.15.53:0/1811129523 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa34f15640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa30103440 0x7faa301a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2f7fe640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa30102a80 0x7faa30199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 -- 172.21.15.53:0/1811129523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa30102a80 msgr2=0x7faa30199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa30102a80 0x7faa30199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 -- 172.21.15.53:0/1811129523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa30103440 msgr2=0x7faa301a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa30103440 0x7faa301a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 -- 172.21.15.53:0/1811129523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa301a0800 con 0x7faa30101890 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2f7fe640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa30102a80 0x7faa30199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa34f15640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa30103440 0x7faa301a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2ffff640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30101890 0x7faa30198be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7faa2000d770 tx=0x7faa2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2d7fa640 1 -- 172.21.15.53:0/1811129523 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20017070 con 0x7faa30101890 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2d7fa640 1 -- 172.21.15.53:0/1811129523 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa20009c30 con 0x7faa30101890 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa301a0af0 con 0x7faa30101890 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.201+0000 7faa2d7fa640 1 -- 172.21.15.53:0/1811129523 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20003cc0 con 0x7faa30101890 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.202+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa301a1040 con 0x7faa30101890 2024-04-28T12:50:09.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.202+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa301a1210 con 0x7faa30101890 2024-04-28T12:50:09.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.204+0000 7faa2d7fa640 1 -- 172.21.15.53:0/1811129523 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faa20003e60 con 0x7faa30101890 2024-04-28T12:50:09.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.204+0000 7faa2d7fa640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa180a3c40 0x7faa180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.204+0000 7faa2f7fe640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa180a3c40 0x7faa180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.204+0000 7faa2d7fa640 1 -- 172.21.15.53:0/1811129523 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7faa200d0410 con 0x7faa30101890 2024-04-28T12:50:09.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.205+0000 7faa2f7fe640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa180a3c40 0x7faa180a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faa3019a0f0 tx=0x7faa24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:09.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.208+0000 7faa2d7fa640 1 -- 172.21.15.53:0/1811129523 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faa20009de0 con 0x7faa30101890 2024-04-28T12:50:09.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.394+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa301a13e0 con 0x7faa30101890 2024-04-28T12:50:09.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.394+0000 7faa2d7fa640 1 -- 172.21.15.53:0/1811129523 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7faa2009cc90 con 0x7faa30101890 2024-04-28T12:50:09.391 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.396+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa180a3c40 msgr2=0x7faa180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.396+0000 7faa36978640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa180a3c40 0x7faa180a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faa3019a0f0 tx=0x7faa24005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.396+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30101890 msgr2=0x7faa30198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.396+0000 7faa36978640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30101890 0x7faa30198be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7faa2000d770 tx=0x7faa2000dc30 comp rx=0 tx=0).stop 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 shutdown_connections 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa180a3c40 0x7faa180a6100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa30103440 0x7faa301a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa30102a80 0x7faa30199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 --2- 172.21.15.53:0/1811129523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa30101890 0x7faa30198be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 >> 172.21.15.53:0/1811129523 conn(0x7faa300fce20 msgr2=0x7faa300fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:09.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 shutdown_connections 2024-04-28T12:50:09.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.397+0000 7faa36978640 1 -- 172.21.15.53:0/1811129523 wait complete. 2024-04-28T12:50:09.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 -- 172.21.15.53:0/3412651086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac103b80 msgr2=0x7f43ac104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 --2- 172.21.15.53:0/3412651086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac103b80 0x7f43ac104000 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f439c009900 tx=0x7f439c02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 -- 172.21.15.53:0/3412651086 shutdown_connections 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 --2- 172.21.15.53:0/3412651086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac104540 0x7f43ac10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 --2- 172.21.15.53:0/3412651086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac103b80 0x7f43ac104000 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 --2- 172.21.15.53:0/3412651086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43ac102990 0x7f43ac102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 -- 172.21.15.53:0/3412651086 >> 172.21.15.53:0/3412651086 conn(0x7f43ac0fdf00 msgr2=0x7f43ac100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 -- 172.21.15.53:0/3412651086 shutdown_connections 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 -- 172.21.15.53:0/3412651086 wait complete. 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.513+0000 7f43b41e2640 1 Processor -- start 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 -- start start 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac102990 0x7f43ac198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac103b80 0x7f43ac199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43ac104540 0x7f43ac1a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac10e4a0 con 0x7f43ac102990 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac10e320 con 0x7f43ac103b80 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac10e620 con 0x7f43ac104540 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac102990 0x7f43ac198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac102990 0x7f43ac198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35988/0 (socket says 172.21.15.53:35988) 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 -- 172.21.15.53:0/138730671 learned_addr learned my addr 172.21.15.53:0/138730671 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b177d640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac103b80 0x7f43ac199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b277f640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43ac104540 0x7f43ac1a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 -- 172.21.15.53:0/138730671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43ac104540 msgr2=0x7f43ac1a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43ac104540 0x7f43ac1a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 -- 172.21.15.53:0/138730671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac103b80 msgr2=0x7f43ac199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac103b80 0x7f43ac199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 -- 172.21.15.53:0/138730671 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac1a07f0 con 0x7f43ac102990 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b177d640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac103b80 0x7f43ac199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b277f640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43ac104540 0x7f43ac1a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b1f7e640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac102990 0x7f43ac198bd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43a800e9c0 tx=0x7f43a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43a2ffd640 1 -- 172.21.15.53:0/138730671 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a800cd80 con 0x7f43ac102990 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43a2ffd640 1 -- 172.21.15.53:0/138730671 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a80035f0 con 0x7f43ac102990 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ac1a0ae0 con 0x7f43ac102990 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.514+0000 7f43a2ffd640 1 -- 172.21.15.53:0/138730671 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8010690 con 0x7f43ac102990 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.515+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ac1a1080 con 0x7f43ac102990 2024-04-28T12:50:09.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.515+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43ac109dd0 con 0x7f43ac102990 2024-04-28T12:50:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.516+0000 7f43a2ffd640 1 -- 172.21.15.53:0/138730671 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f43a80037a0 con 0x7f43ac102990 2024-04-28T12:50:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.517+0000 7f43a2ffd640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f43980a3cc0 0x7f43980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.517+0000 7f43b177d640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f43980a3cc0 0x7f43980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.517+0000 7f43a2ffd640 1 -- 172.21.15.53:0/138730671 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f43a80d09f0 con 0x7f43ac102990 2024-04-28T12:50:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.518+0000 7f43b177d640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f43980a3cc0 0x7f43980a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f439c006010 tx=0x7f439c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.522+0000 7f43a2ffd640 1 -- 172.21.15.53:0/138730671 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f43a809d270 con 0x7f43ac102990 2024-04-28T12:50:09.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.707+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f43ac02dad0 con 0x7f43ac102990 2024-04-28T12:50:09.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.708+0000 7f43a2ffd640 1 -- 172.21.15.53:0/138730671 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f43ac02dad0 con 0x7f43ac102990 2024-04-28T12:50:09.704 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-04-28T12:50:09.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f43980a3cc0 msgr2=0x7f43980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f43980a3cc0 0x7f43980a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f439c006010 tx=0x7f439c000bc0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac102990 msgr2=0x7f43ac198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac102990 0x7f43ac198bd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43a800e9c0 tx=0x7f43a800ee80 comp rx=0 tx=0).stop 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 shutdown_connections 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f43980a3cc0 0x7f43980a6180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43ac104540 0x7f43ac1a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43ac103b80 0x7f43ac199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 --2- 172.21.15.53:0/138730671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43ac102990 0x7f43ac198bd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 >> 172.21.15.53:0/138730671 conn(0x7f43ac0fdf00 msgr2=0x7f43ac106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 shutdown_connections 2024-04-28T12:50:09.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.710+0000 7f43b41e2640 1 -- 172.21.15.53:0/138730671 wait complete. 2024-04-28T12:50:09.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:50:09.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 -- 172.21.15.53:0/669312467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc102a80 msgr2=0x7f89dc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 --2- 172.21.15.53:0/669312467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc102a80 0x7f89dc102f00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009900 tx=0x7f89cc02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:09.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 -- 172.21.15.53:0/669312467 shutdown_connections 2024-04-28T12:50:09.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 --2- 172.21.15.53:0/669312467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc103440 0x7f89dc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 --2- 172.21.15.53:0/669312467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc102a80 0x7f89dc102f00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 --2- 172.21.15.53:0/669312467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89dc101890 0x7f89dc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 -- 172.21.15.53:0/669312467 >> 172.21.15.53:0/669312467 conn(0x7f89dc0fce20 msgr2=0x7f89dc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 -- 172.21.15.53:0/669312467 shutdown_connections 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 -- 172.21.15.53:0/669312467 wait complete. 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 Processor -- start 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.826+0000 7f89e1293640 1 -- start start 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89e1293640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc101890 0x7f89dc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89e1293640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89dc102a80 0x7f89dc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89e1293640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc103440 0x7f89dc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89e1293640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10d3c0 con 0x7f89dc101890 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89e1293640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10d240 con 0x7f89dc103440 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89e1293640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10d540 con 0x7f89dc102a80 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc101890 0x7f89dc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc101890 0x7f89dc197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36004/0 (socket says 172.21.15.53:36004) 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 -- 172.21.15.53:0/3743183427 learned_addr learned my addr 172.21.15.53:0/3743183427 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89db7fe640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc103440 0x7f89dc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89da7fc640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89dc102a80 0x7f89dc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 -- 172.21.15.53:0/3743183427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89dc102a80 msgr2=0x7f89dc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89dc102a80 0x7f89dc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 -- 172.21.15.53:0/3743183427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc103440 msgr2=0x7f89dc19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc103440 0x7f89dc19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 -- 172.21.15.53:0/3743183427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc19f6f0 con 0x7f89dc101890 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89da7fc640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89dc102a80 0x7f89dc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89daffd640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc101890 0x7f89dc197ad0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f89c400d950 tx=0x7f89c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89d3fff640 1 -- 172.21.15.53:0/3743183427 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4017070 con 0x7f89dc101890 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89d3fff640 1 -- 172.21.15.53:0/3743183427 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c4009c30 con 0x7f89dc101890 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc19f9e0 con 0x7f89dc101890 2024-04-28T12:50:09.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.827+0000 7f89d3fff640 1 -- 172.21.15.53:0/3743183427 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c400f620 con 0x7f89dc101890 2024-04-28T12:50:09.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.828+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc19ff30 con 0x7f89dc101890 2024-04-28T12:50:09.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.828+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc108ca0 con 0x7f89dc101890 2024-04-28T12:50:09.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.830+0000 7f89d3fff640 1 -- 172.21.15.53:0/3743183427 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f89c401a070 con 0x7f89dc101890 2024-04-28T12:50:09.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.830+0000 7f89d3fff640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f89c80a3cc0 0x7f89c80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:09.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.830+0000 7f89da7fc640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f89c80a3cc0 0x7f89c80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:09.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.830+0000 7f89d3fff640 1 -- 172.21.15.53:0/3743183427 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f89c40cff70 con 0x7f89dc101890 2024-04-28T12:50:09.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.831+0000 7f89da7fc640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f89c80a3cc0 0x7f89c80a6180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f89dc198fe0 tx=0x7f89cc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:09.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:09.835+0000 7f89d3fff640 1 -- 172.21.15.53:0/3743183427 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f89c4009de0 con 0x7f89dc101890 2024-04-28T12:50:09.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:09 smithi077 ceph-mon[32553]: pgmap v335: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:09.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:09 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/560076947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:09.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:09 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1600189698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:09.948 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:09 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1811129523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:10.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.020+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f89dc063530 con 0x7f89dc101890 2024-04-28T12:50:10.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.020+0000 7f89d3fff640 1 -- 172.21.15.53:0/3743183427 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f89c409c7f0 con 0x7f89dc101890 2024-04-28T12:50:10.017 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f89c80a3cc0 msgr2=0x7f89c80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f89c80a3cc0 0x7f89c80a6180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f89dc198fe0 tx=0x7f89cc0314a0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc101890 msgr2=0x7f89dc197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc101890 0x7f89dc197ad0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f89c400d950 tx=0x7f89c400de10 comp rx=0 tx=0).stop 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 shutdown_connections 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f89c80a3cc0 0x7f89c80a6180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc103440 0x7f89dc19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89dc102a80 0x7f89dc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 --2- 172.21.15.53:0/3743183427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89dc101890 0x7f89dc197ad0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 >> 172.21.15.53:0/3743183427 conn(0x7f89dc0fce20 msgr2=0x7f89dc105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.022+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 shutdown_connections 2024-04-28T12:50:10.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.023+0000 7f89e1293640 1 -- 172.21.15.53:0/3743183427 wait complete. 2024-04-28T12:50:10.029 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:09 smithi053 ceph-mon[29182]: pgmap v335: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:10.029 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:09 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/560076947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:10.029 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:09 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1600189698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:10.029 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:09 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1811129523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:10.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:09 smithi191 ceph-mon[33016]: pgmap v335: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:09 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/560076947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:09 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1600189698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:09 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1811129523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.135+0000 7f5b8256a640 1 -- 172.21.15.53:0/1330355017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c103420 msgr2=0x7f5b7c109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.135+0000 7f5b8256a640 1 --2- 172.21.15.53:0/1330355017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c103420 0x7f5b7c109cb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009ca0 tx=0x7f5b7402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 -- 172.21.15.53:0/1330355017 shutdown_connections 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 --2- 172.21.15.53:0/1330355017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c103420 0x7f5b7c109cb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 --2- 172.21.15.53:0/1330355017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c102a60 0x7f5b7c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 --2- 172.21.15.53:0/1330355017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b7c101870 0x7f5b7c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 -- 172.21.15.53:0/1330355017 >> 172.21.15.53:0/1330355017 conn(0x7f5b7c0fcde0 msgr2=0x7f5b7c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:10.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 -- 172.21.15.53:0/1330355017 shutdown_connections 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 -- 172.21.15.53:0/1330355017 wait complete. 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 Processor -- start 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 -- start start 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c101870 0x7f5b7c06a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b7c102a60 0x7f5b7c19d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.136+0000 7f5b8256a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c103420 0x7f5b7c19fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b8256a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10d3a0 con 0x7f5b7c101870 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b8256a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10d220 con 0x7f5b7c103420 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b8256a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10d520 con 0x7f5b7c102a60 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c101870 0x7f5b7c06a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c101870 0x7f5b7c06a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36024/0 (socket says 172.21.15.53:36024) 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 -- 172.21.15.53:0/974132966 learned_addr learned my addr 172.21.15.53:0/974132966 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b80b07640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c103420 0x7f5b7c19fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7b7fe640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b7c102a60 0x7f5b7c19d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 -- 172.21.15.53:0/974132966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b7c102a60 msgr2=0x7f5b7c19d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b7c102a60 0x7f5b7c19d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 -- 172.21.15.53:0/974132966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c103420 msgr2=0x7f5b7c19fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c103420 0x7f5b7c19fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 -- 172.21.15.53:0/974132966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7c1a0760 con 0x7f5b7c101870 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b7bfff640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c101870 0x7f5b7c06a690 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9c0 tx=0x7f5b6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b797fa640 1 -- 172.21.15.53:0/974132966 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400cd80 con 0x7f5b7c101870 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b797fa640 1 -- 172.21.15.53:0/974132966 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b640035f0 con 0x7f5b7c101870 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7c1a0a50 con 0x7f5b7c101870 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b797fa640 1 -- 172.21.15.53:0/974132966 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b64010690 con 0x7f5b7c101870 2024-04-28T12:50:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.137+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b7c1a0f50 con 0x7f5b7c101870 2024-04-28T12:50:10.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.138+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b7c108c10 con 0x7f5b7c101870 2024-04-28T12:50:10.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.139+0000 7f5b797fa640 1 -- 172.21.15.53:0/974132966 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5b64004000 con 0x7f5b7c101870 2024-04-28T12:50:10.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.140+0000 7f5b797fa640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5b680a3c40 0x7f5b680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.140+0000 7f5b7b7fe640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5b680a3c40 0x7f5b680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.140+0000 7f5b797fa640 1 -- 172.21.15.53:0/974132966 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f5b640d0b30 con 0x7f5b7c101870 2024-04-28T12:50:10.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.140+0000 7f5b7b7fe640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5b680a3c40 0x7f5b680a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c06a400 tx=0x7f5b6c00b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:10.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.144+0000 7f5b797fa640 1 -- 172.21.15.53:0/974132966 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5b6409d3b0 con 0x7f5b7c101870 2024-04-28T12:50:10.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.370+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b7c02dad0 con 0x7f5b7c101870 2024-04-28T12:50:10.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.371+0000 7f5b797fa640 1 -- 172.21.15.53:0/974132966 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11873 (secure 0 0 0) 0x7f5b6409ccb0 con 0x7f5b7c101870 2024-04-28T12:50:10.367 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:10.367 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:48:49.455677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24485,24478]},"id":1}]} 2024-04-28T12:50:10.368 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 22 2024-04-28T12:50:10.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5b680a3c40 msgr2=0x7f5b680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5b680a3c40 0x7f5b680a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c06a400 tx=0x7f5b6c00b590 comp rx=0 tx=0).stop 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c101870 msgr2=0x7f5b7c06a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c101870 0x7f5b7c06a690 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9c0 tx=0x7f5b6400ee80 comp rx=0 tx=0).stop 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 shutdown_connections 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5b680a3c40 0x7f5b680a6100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c103420 0x7f5b7c19fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b7c102a60 0x7f5b7c19d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 --2- 172.21.15.53:0/974132966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b7c101870 0x7f5b7c06a690 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 >> 172.21.15.53:0/974132966 conn(0x7f5b7c0fcde0 msgr2=0x7f5b7c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 shutdown_connections 2024-04-28T12:50:10.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.373+0000 7f5b8256a640 1 -- 172.21.15.53:0/974132966 wait complete. 2024-04-28T12:50:10.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 -- 172.21.15.53:0/1098358713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 msgr2=0x7fcf30102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/1098358713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30102ee0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009900 tx=0x7fcf1c02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 -- 172.21.15.53:0/1098358713 shutdown_connections 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/1098358713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf30103420 0x7fcf30109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/1098358713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30102ee0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/1098358713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf30101870 0x7fcf30101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 -- 172.21.15.53:0/1098358713 >> 172.21.15.53:0/1098358713 conn(0x7fcf300fcde0 msgr2=0x7fcf300ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 -- 172.21.15.53:0/1098358713 shutdown_connections 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.486+0000 7fcf36cbb640 1 -- 172.21.15.53:0/1098358713 wait complete. 2024-04-28T12:50:10.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 Processor -- start 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 -- start start 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf30101870 0x7fcf30198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf30103420 0x7fcf301a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3010e500 con 0x7fcf30102a60 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3010e380 con 0x7fcf30103420 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf36cbb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3010e680 con 0x7fcf30101870 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf2ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf2ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36056/0 (socket says 172.21.15.53:36056) 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf2ffff640 1 -- 172.21.15.53:0/34343025 learned_addr learned my addr 172.21.15.53:0/34343025 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.487+0000 7fcf34a57640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf30101870 0x7fcf30198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2ffff640 1 -- 172.21.15.53:0/34343025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf30101870 msgr2=0x7fcf30198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2ffff640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf30101870 0x7fcf30198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf35258640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf30103420 0x7fcf301a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2ffff640 1 -- 172.21.15.53:0/34343025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf30103420 msgr2=0x7fcf301a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2ffff640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf30103420 0x7fcf301a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2ffff640 1 -- 172.21.15.53:0/34343025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf301a0830 con 0x7fcf30102a60 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf34a57640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf30101870 0x7fcf30198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf35258640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf30103420 0x7fcf301a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2ffff640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30199160 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c005940 tx=0x7fcf1c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2dffb640 1 -- 172.21.15.53:0/34343025 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c03f030 con 0x7fcf30102a60 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf301a0ac0 con 0x7fcf30102a60 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2dffb640 1 -- 172.21.15.53:0/34343025 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf1c003cb0 con 0x7fcf30102a60 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf2dffb640 1 -- 172.21.15.53:0/34343025 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c035070 con 0x7fcf30102a60 2024-04-28T12:50:10.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.488+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf301a1000 con 0x7fcf30102a60 2024-04-28T12:50:10.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.489+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf30108be0 con 0x7fcf30102a60 2024-04-28T12:50:10.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.491+0000 7fcf2dffb640 1 -- 172.21.15.53:0/34343025 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcf1c042030 con 0x7fcf30102a60 2024-04-28T12:50:10.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.492+0000 7fcf2dffb640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcf200a3c40 0x7fcf200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.492+0000 7fcf34a57640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcf200a3c40 0x7fcf200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.492+0000 7fcf2dffb640 1 -- 172.21.15.53:0/34343025 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fcf1c0f1280 con 0x7fcf30102a60 2024-04-28T12:50:10.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.492+0000 7fcf34a57640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcf200a3c40 0x7fcf200a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcf3019a0f0 tx=0x7fcf24005e90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:10.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.495+0000 7fcf2dffb640 1 -- 172.21.15.53:0/34343025 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcf1c0bdac0 con 0x7fcf30102a60 2024-04-28T12:50:10.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.683+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf3002dc50 con 0x7fcf30102a60 2024-04-28T12:50:10.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.684+0000 7fcf2dffb640 1 -- 172.21.15.53:0/34343025 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7fcf1c003e60 con 0x7fcf30102a60 2024-04-28T12:50:10.681 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:10.681 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcf200a3c40 msgr2=0x7fcf200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcf200a3c40 0x7fcf200a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcf3019a0f0 tx=0x7fcf24005e90 comp rx=0 tx=0).stop 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 msgr2=0x7fcf30199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30199160 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c005940 tx=0x7fcf1c004430 comp rx=0 tx=0).stop 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 shutdown_connections 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcf200a3c40 0x7fcf200a6100 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf30103420 0x7fcf301a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf30102a60 0x7fcf30199160 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 --2- 172.21.15.53:0/34343025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf30101870 0x7fcf30198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.686+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 >> 172.21.15.53:0/34343025 conn(0x7fcf300fcde0 msgr2=0x7fcf30105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.687+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 shutdown_connections 2024-04-28T12:50:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.687+0000 7fcf36cbb640 1 -- 172.21.15.53:0/34343025 wait complete. 2024-04-28T12:50:10.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 -- 172.21.15.53:0/2114527403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c103420 msgr2=0x7fc31c109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 --2- 172.21.15.53:0/2114527403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c103420 0x7fc31c109cb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc314009ca0 tx=0x7fc31402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 -- 172.21.15.53:0/2114527403 shutdown_connections 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 --2- 172.21.15.53:0/2114527403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c103420 0x7fc31c109cb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 --2- 172.21.15.53:0/2114527403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31c102a60 0x7fc31c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 --2- 172.21.15.53:0/2114527403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc31c101870 0x7fc31c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 -- 172.21.15.53:0/2114527403 >> 172.21.15.53:0/2114527403 conn(0x7fc31c0fcde0 msgr2=0x7fc31c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 -- 172.21.15.53:0/2114527403 shutdown_connections 2024-04-28T12:50:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.801+0000 7fc321893640 1 -- 172.21.15.53:0/2114527403 wait complete. 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 Processor -- start 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 -- start start 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c101870 0x7fc31c06a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31c102a60 0x7fc31c19d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc31c103420 0x7fc31c19fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c10d3a0 con 0x7fc31c101870 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c10d220 con 0x7fc31c102a60 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc321893640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c10d520 con 0x7fc31c103420 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c101870 0x7fc31c06a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c101870 0x7fc31c06a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36082/0 (socket says 172.21.15.53:36082) 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 -- 172.21.15.53:0/740462285 learned_addr learned my addr 172.21.15.53:0/740462285 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc313fff640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31c102a60 0x7fc31c19d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31b7fe640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc31c103420 0x7fc31c19fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 -- 172.21.15.53:0/740462285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc31c103420 msgr2=0x7fc31c19fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc31c103420 0x7fc31c19fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 -- 172.21.15.53:0/740462285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31c102a60 msgr2=0x7fc31c19d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31c102a60 0x7fc31c19d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.802+0000 7fc31affd640 1 -- 172.21.15.53:0/740462285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c1a0760 con 0x7fc31c101870 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.803+0000 7fc31affd640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c101870 0x7fc31c06a690 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00d950 tx=0x7fc30c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.803+0000 7fc320891640 1 -- 172.21.15.53:0/740462285 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c017070 con 0x7fc31c101870 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.803+0000 7fc320891640 1 -- 172.21.15.53:0/740462285 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c009c30 con 0x7fc31c101870 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.803+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31c1a0a50 con 0x7fc31c101870 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.803+0000 7fc320891640 1 -- 172.21.15.53:0/740462285 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c00f620 con 0x7fc31c101870 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.803+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31c1a0fa0 con 0x7fc31c101870 2024-04-28T12:50:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.804+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc31c108c10 con 0x7fc31c101870 2024-04-28T12:50:10.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.805+0000 7fc320891640 1 -- 172.21.15.53:0/740462285 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc30c01a070 con 0x7fc31c101870 2024-04-28T12:50:10.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.805+0000 7fc320891640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc3080a3cc0 0x7fc3080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:10.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.806+0000 7fc313fff640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc3080a3cc0 0x7fc3080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:10.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.806+0000 7fc320891640 1 -- 172.21.15.53:0/740462285 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fc30c0cff70 con 0x7fc31c101870 2024-04-28T12:50:10.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.806+0000 7fc313fff640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc3080a3cc0 0x7fc3080a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc31c06a400 tx=0x7fc304005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:10.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:10.810+0000 7fc320891640 1 -- 172.21.15.53:0/740462285 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc30c009de0 con 0x7fc31c101870 2024-04-28T12:50:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.039+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc31c02dad0 con 0x7fc31c101870 2024-04-28T12:50:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.040+0000 7fc320891640 1 -- 172.21.15.53:0/740462285 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11873 (secure 0 0 0) 0x7fc30c09c7f0 con 0x7fc31c101870 2024-04-28T12:50:11.037 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:11.037 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:48:49.455677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24485,24478]},"id":1}]} 2024-04-28T12:50:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 22 2024-04-28T12:50:11.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:10 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/138730671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:11.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:10 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3743183427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:11.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:10 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/974132966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:11.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:10 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/34343025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc3080a3cc0 msgr2=0x7fc3080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc3080a3cc0 0x7fc3080a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc31c06a400 tx=0x7fc304005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c101870 msgr2=0x7fc31c06a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c101870 0x7fc31c06a690 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00d950 tx=0x7fc30c00de10 comp rx=0 tx=0).stop 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 shutdown_connections 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fc3080a3cc0 0x7fc3080a6180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc31c103420 0x7fc31c19fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31c102a60 0x7fc31c19d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 --2- 172.21.15.53:0/740462285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc31c101870 0x7fc31c06a690 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 >> 172.21.15.53:0/740462285 conn(0x7fc31c0fcde0 msgr2=0x7fc31c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 shutdown_connections 2024-04-28T12:50:11.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.042+0000 7fc321893640 1 -- 172.21.15.53:0/740462285 wait complete. 2024-04-28T12:50:11.049 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:10 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/138730671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:11.049 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:10 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3743183427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:11.049 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:10 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/974132966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:11.049 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:10 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/34343025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:11.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-28T12:48:43.873799+0000', 'modified': '2024-04-28T12:48:49.455677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24505, 'mds_1': 24485, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6833/3944921810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 3944921810}, {'type': 'v1', 'addr': '172.21.15.77:6833', 'nonce': 3944921810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6835/1670611904', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 1670611904}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 1670611904}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/2819920278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 2819920278}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 2819920278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.191:6837/4132012864', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 4132012864}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 4132012864}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.77:6837/2147382100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2147382100}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2147382100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6839/2399731075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 2399731075}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 2399731075}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24505, 'qdb_cluster': [24505, 24485, 24478]} 2024-04-28T12:50:11.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-28T12:50:11.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 -- 172.21.15.53:0/1902447616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8102a60 msgr2=0x7ff3d8102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 --2- 172.21.15.53:0/1902447616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8102a60 0x7ff3d8102ee0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009900 tx=0x7ff3c002a410 comp rx=0 tx=0).stop 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 -- 172.21.15.53:0/1902447616 shutdown_connections 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 --2- 172.21.15.53:0/1902447616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3d8103420 0x7ff3d8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 --2- 172.21.15.53:0/1902447616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8102a60 0x7ff3d8102ee0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 --2- 172.21.15.53:0/1902447616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3d8101870 0x7ff3d8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 -- 172.21.15.53:0/1902447616 >> 172.21.15.53:0/1902447616 conn(0x7ff3d80fcde0 msgr2=0x7ff3d80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 -- 172.21.15.53:0/1902447616 shutdown_connections 2024-04-28T12:50:11.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.157+0000 7ff3dd712640 1 -- 172.21.15.53:0/1902447616 wait complete. 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 Processor -- start 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 -- start start 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3d8101870 0x7ff3d819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3d8102a60 0x7ff3d819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8103420 0x7ff3d81a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d810e4c0 con 0x7ff3d8103420 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d810e340 con 0x7ff3d8102a60 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3dd712640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d810e640 con 0x7ff3d8101870 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3d77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8103420 0x7ff3d81a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3d77fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8103420 0x7ff3d81a2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36094/0 (socket says 172.21.15.53:36094) 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3d77fe640 1 -- 172.21.15.53:0/576338105 learned_addr learned my addr 172.21.15.53:0/576338105 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.158+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3d8101870 0x7ff3d819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3d8102a60 0x7ff3d819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d77fe640 1 -- 172.21.15.53:0/576338105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3d8101870 msgr2=0x7ff3d819acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d77fe640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3d8101870 0x7ff3d819acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d77fe640 1 -- 172.21.15.53:0/576338105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3d8102a60 msgr2=0x7ff3d819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d77fe640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3d8102a60 0x7ff3d819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d77fe640 1 -- 172.21.15.53:0/576338105 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d81a2900 con 0x7ff3d8103420 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3d8101870 0x7ff3d819acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3d8102a60 0x7ff3d819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3d77fe640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8103420 0x7ff3d81a2260 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000c970 tx=0x7ff3d000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3cffff640 1 -- 172.21.15.53:0/576338105 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0007bb0 con 0x7ff3d8103420 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d81a2bf0 con 0x7ff3d8103420 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3cffff640 1 -- 172.21.15.53:0/576338105 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3d0007d50 con 0x7ff3d8103420 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3cffff640 1 -- 172.21.15.53:0/576338105 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0018770 con 0x7ff3d8103420 2024-04-28T12:50:11.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.159+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d81a3140 con 0x7ff3d8103420 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.161+0000 7ff3cffff640 1 -- 172.21.15.53:0/576338105 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff3d000f990 con 0x7ff3d8103420 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.161+0000 7ff3e042a740 1 Processor -- start 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.161+0000 7ff3e042a740 1 -- start start 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a12ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a17b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a1a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3e042a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564570977740 con 0x564570a12690 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3e042a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5645709133a0 con 0x564570a13080 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3e042a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564570973700 con 0x564570a181f0 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a12ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a12ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36110/0 (socket says 172.21.15.53:36110) 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 -- 172.21.15.53:0/844166850 learned_addr learned my addr 172.21.15.53:0/844166850 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3cffff640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff3c40a3c40 0x7ff3c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a17b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 -- 172.21.15.53:0/844166850 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 msgr2=0x564570a1a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a1a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 -- 172.21.15.53:0/844166850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 msgr2=0x564570a17b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a17b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d77fe640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff3c40a3c40 0x7ff3c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 -- 172.21.15.53:0/844166850 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56457092f750 con 0x564570a12690 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3cffff640 1 -- 172.21.15.53:0/576338105 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7ff3d00d07f0 con 0x7ff3d8103420 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.162+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a12ab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0006010 tx=0x7ff3c0032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c26c99b3c619a73 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3cd7fa640 1 -- 172.21.15.53:0/844166850 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0040070 con 0x564570a12690 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3d77fe640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff3c40a3c40 0x7ff3c40a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0099270 tx=0x7ff3d00057f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3cd7fa640 1 -- 172.21.15.53:0/844166850 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3c0031860 con 0x564570a12690 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3cd7fa640 1 -- 172.21.15.53:0/844166850 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0035050 con 0x564570a12690 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 -- 172.21.15.53:0/844166850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 msgr2=0x564570a12ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a12ab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0006010 tx=0x7ff3c0032d60 comp rx=0 tx=0).stop 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 -- 172.21.15.53:0/844166850 shutdown_connections 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a1a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a17b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 --2- 172.21.15.53:0/844166850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a12ab0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 -- 172.21.15.53:0/844166850 >> 172.21.15.53:0/844166850 conn(0x564570a0dd00 msgr2=0x564570a0e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.163+0000 7ff3e042a740 1 -- 172.21.15.53:0/844166850 shutdown_connections 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 -- 172.21.15.53:0/844166850 wait complete. 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 -- start start 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 Processor -- start 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a13080 0x564570a1d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a24060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564570a1b140 con 0x564570a13080 2024-04-28T12:50:11.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5645708fefe0 con 0x564570a12690 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564570a1b2c0 con 0x564570a181f0 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3e042a740 10 client.0 _ll_register_callbacks cb 0x564570915150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a13080 0x564570a1d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a13080 0x564570a1d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36112/0 (socket says 172.21.15.53:36112) 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 -- 172.21.15.53:0/1938296580 learned_addr learned my addr 172.21.15.53:0/1938296580 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a24060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 -- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 msgr2=0x564570a24060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a24060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 -- 172.21.15.53:0/1938296580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 msgr2=0x564570a1cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d6ffd640 1 -- 172.21.15.53:0/1938296580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645706892d0 con 0x564570a13080 2024-04-28T12:50:11.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.164+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a13080 0x564570a1d020 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8010a40 tx=0x7ff3c8010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e3bf45a0b681ad server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1938296580 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8004d20 con 0x564570a13080 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c0031860 con 0x564570a13080 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1938296580 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7ff3c800ea50 con 0x564570a13080 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1938296580 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff3c800ec50 con 0x564570a13080 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 10 client.15054 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5645707536b0 con 0x564570a13080 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1938296580 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3c0031860 con 0x564570a13080 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1938296580 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff3c8004460 con 0x564570a13080 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 10 client.15054 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 10 client.15054 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 20 client.15054 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:11.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a2aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 4 client.15054 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.165+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 --> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564570985350 con 0x564570a295f0 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.166+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a2aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.166+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a2aa60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a8f38bf5775501 server_cookie=3ff62f5b776647db in_seq=0 out_seq=0 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.166+0000 7ff3be7fc640 10 client.15054 ms_handle_connect on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1938296580 <== mds.2 v2:172.21.15.191:6836/4132012864 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564570985350 con 0x564570a295f0 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3be7fc640 10 client.15054 handle_command_reply: tid=0 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 client.15054 shutdown 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a13080 msgr2=0x564570a1d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a13080 0x564570a1d020 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8010a40 tx=0x7ff3c8010f00 comp rx=0 tx=0).stop 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 shutdown_connections 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a2aa60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a24060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3be7fc640 0 client.15054 ms_handle_reset on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:11.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a13080 0x564570a1d020 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1938296580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.167+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 >> 172.21.15.53:0/1938296580 conn(0x564570a0dd00 msgr2=0x564570a0cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.168+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 shutdown_connections 2024-04-28T12:50:11.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.168+0000 7ff3e042a740 1 -- 172.21.15.53:0/1938296580 wait complete. 2024-04-28T12:50:11.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.168+0000 7ff3e042a740 20 client.15054 trim_cache size 0 max 16384 2024-04-28T12:50:11.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 Processor -- start 2024-04-28T12:50:11.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 -- start start 2024-04-28T12:50:11.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a13080 0x564570a24020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a181f0 0x564570a1cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564570a28da0 con 0x564570a181f0 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564570a29470 con 0x564570a12690 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3e042a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564570a29b40 con 0x564570a13080 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a181f0 0x564570a1cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a181f0 0x564570a1cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36114/0 (socket says 172.21.15.53:36114) 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1087380854 learned_addr learned my addr 172.21.15.53:0/1087380854 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a13080 0x564570a24020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1087380854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a13080 msgr2=0x564570a24020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a13080 0x564570a24020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1087380854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 msgr2=0x564570a1c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1087380854 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564570a0d840 con 0x564570a181f0 2024-04-28T12:50:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.186+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a13080 0x564570a24020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.187+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a181f0 0x564570a1cc30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x564570a138e0 tx=0x7ff3c002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89456bdc3471772 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.187+0000 7ff3ccff9640 1 -- 172.21.15.53:0/1087380854 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0057070 con 0x564570a181f0 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.187+0000 7ff3ccff9640 1 -- 172.21.15.53:0/1087380854 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3c0031660 con 0x564570a181f0 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.187+0000 7ff3ccff9640 1 -- 172.21.15.53:0/1087380854 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0035040 con 0x564570a181f0 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.187+0000 7ff3e042a740 1 -- 172.21.15.53:0/1087380854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a181f0 msgr2=0x564570a1cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.187+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a181f0 0x564570a1cc30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x564570a138e0 tx=0x7ff3c002a410 comp rx=0 tx=0).stop 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.187+0000 7ff3e042a740 1 -- 172.21.15.53:0/1087380854 shutdown_connections 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a181f0 0x564570a1cc30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a13080 0x564570a24020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1087380854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a12690 0x564570a1c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 -- 172.21.15.53:0/1087380854 >> 172.21.15.53:0/1087380854 conn(0x564570a0dd00 msgr2=0x564570a14480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 -- 172.21.15.53:0/1087380854 shutdown_connections 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 -- 172.21.15.53:0/1087380854 wait complete. 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 -- start start 2024-04-28T12:50:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 Processor -- start 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a1fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a20340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a23360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x564570a2a8e0 con 0x564570a12690 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564570a2a210 con 0x564570a13080 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564570a2afb0 con 0x564570a181f0 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3e042a740 10 client.0 _ll_register_callbacks cb 0x7ff3c8007660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3d67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a1fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3d67fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a1fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36116/0 (socket says 172.21.15.53:36116) 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1578255503 learned_addr learned my addr 172.21.15.53:0/1578255503 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.188+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a20340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1578255503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 msgr2=0x564570a23360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a23360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1578255503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 msgr2=0x564570a20340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a20340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3d67fc640 1 -- 172.21.15.53:0/1578255503 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564570a0ea70 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3d67fc640 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a1fe00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0005350 tx=0x7ff3c0005380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebb50fa5c964b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1578255503 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0057070 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1578255503 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7ff3c0031d10 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c0031660 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1578255503 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff3c0035030 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3e042a740 10 client.15060 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564570a8f9f0 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1578255503 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3c0031660 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1578255503 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ff3c0003680 con 0x564570a12690 2024-04-28T12:50:11.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3e042a740 10 client.15060 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.189+0000 7ff3e042a740 10 client.15060 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.190+0000 7ff3e042a740 20 client.15060 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.190+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a92cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.190+0000 7ff3e042a740 4 client.15060 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.190+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 --> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] -- command(tid 0: {"prefix": "status"}) -- 0x56457097f1c0 con 0x564570a295f0 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.190+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a92cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.190+0000 7ff3be7fc640 10 client.15060 ms_handle_connect on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.190+0000 7ff3d6ffd640 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a92cd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d52e0ef630ff376 server_cookie=9af4374042e064a2 in_seq=0 out_seq=0 2024-04-28T12:50:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3be7fc640 1 -- 172.21.15.53:0/1578255503 <== mds.2 v2:172.21.15.191:6836/4132012864 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56457097f1c0 con 0x564570a295f0 2024-04-28T12:50:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3be7fc640 10 client.15060 handle_command_reply: tid=0 2024-04-28T12:50:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 client.15060 shutdown 2024-04-28T12:50:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 msgr2=0x564570a1fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a1fe00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0005350 tx=0x7ff3c0005380 comp rx=0 tx=0).stop 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 shutdown_connections 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x564570a295f0 0x564570a92cd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564570a181f0 0x564570a23360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3be7fc640 0 client.15060 ms_handle_reset on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564570a13080 0x564570a20340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 --2- 172.21.15.53:0/1578255503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564570a12690 0x564570a1fe00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 >> 172.21.15.53:0/1578255503 conn(0x564570a0dd00 msgr2=0x564570a1c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 shutdown_connections 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 1 -- 172.21.15.53:0/1578255503 wait complete. 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3e042a740 20 client.15060 trim_cache size 0 max 16384 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-28T12:50:11.193 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 2, 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24478, 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 82.649102980999999, 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 22, 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 88, 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 86, 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 107.170220038 2024-04-28T12:50:11.194 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-28T12:50:11.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff3c40a3c40 msgr2=0x7ff3c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3dd712640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff3c40a3c40 0x7ff3c40a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0099270 tx=0x7ff3d00057f0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8103420 msgr2=0x7ff3d81a2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.191+0000 7ff3dd712640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8103420 0x7ff3d81a2260 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000c970 tx=0x7ff3d000ce30 comp rx=0 tx=0).stop 2024-04-28T12:50:11.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 shutdown_connections 2024-04-28T12:50:11.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff3c40a3c40 0x7ff3c40a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3d8103420 0x7ff3d81a2260 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3d8102a60 0x7ff3d819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 --2- 172.21.15.53:0/576338105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3d8101870 0x7ff3d819acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 >> 172.21.15.53:0/576338105 conn(0x7ff3d80fcde0 msgr2=0x7ff3d810a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 shutdown_connections 2024-04-28T12:50:11.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.192+0000 7ff3dd712640 1 -- 172.21.15.53:0/576338105 wait complete. 2024-04-28T12:50:11.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:10 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/138730671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:11.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:10 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3743183427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:11.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:10 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/974132966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:11.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:10 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/34343025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:11.199 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2342651693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 msgr2=0x7f5cdc10aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2342651693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc10aac0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009ca0 tx=0x7f5cd402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2342651693 shutdown_connections 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2342651693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc10aac0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2342651693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cdc105da0 0x7f5cdc108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2342651693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cdc103470 0x7f5cdc105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2342651693 >> 172.21.15.53:0/2342651693 conn(0x7f5cdc0fce20 msgr2=0x7f5cdc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2342651693 shutdown_connections 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.305+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2342651693 wait complete. 2024-04-28T12:50:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 Processor -- start 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 -- start start 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cdc103470 0x7f5cdc198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cdc105da0 0x7f5cdc199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc1a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc10e4c0 con 0x7f5cdc1086d0 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc10e340 con 0x7f5cdc103470 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce30c8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc10e640 con 0x7f5cdc105da0 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce1665640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc1a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce1665640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc1a00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36136/0 (socket says 172.21.15.53:36136) 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.306+0000 7f5ce1665640 1 -- 172.21.15.53:0/2175807025 learned_addr learned my addr 172.21.15.53:0/2175807025 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cdc103470 0x7f5cdc198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cdc105da0 0x7f5cdc199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce1665640 1 -- 172.21.15.53:0/2175807025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cdc105da0 msgr2=0x7f5cdc199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce1665640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cdc105da0 0x7f5cdc199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce1665640 1 -- 172.21.15.53:0/2175807025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cdc103470 msgr2=0x7f5cdc198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce1665640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cdc103470 0x7f5cdc198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce1665640 1 -- 172.21.15.53:0/2175807025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc1a0720 con 0x7f5cdc1086d0 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cdc105da0 0x7f5cdc199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cdc103470 0x7f5cdc198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce1665640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc1a00b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097b0 tx=0x7f5cd4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5cd9ffb640 1 -- 172.21.15.53:0/2175807025 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4004240 con 0x7f5cdc1086d0 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5cd9ffb640 1 -- 172.21.15.53:0/2175807025 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cd40317c0 con 0x7f5cdc1086d0 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cdc1a09b0 con 0x7f5cdc1086d0 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5cd9ffb640 1 -- 172.21.15.53:0/2175807025 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd402fd80 con 0x7f5cdc1086d0 2024-04-28T12:50:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.307+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cdc1a0f50 con 0x7f5cdc1086d0 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5cd9ffb640 1 -- 172.21.15.53:0/2175807025 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5cd40043e0 con 0x7f5cdc1086d0 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 Processor -- start 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 -- start start 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2ba080 0x55f58c2beb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5cd9ffb640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5ccc0a3cc0 0x7f5ccc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f58c21e740 con 0x55f58c2bf1f0 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f58c1ba3a0 con 0x55f58c2ba080 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f58c21a700 con 0x55f58c2b9690 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce1665640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5ccc0a3cc0 0x7f5ccc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce1665640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36146/0 (socket says 172.21.15.53:36146) 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.310+0000 7f5ce1665640 1 -- 172.21.15.53:0/901023294 learned_addr learned my addr 172.21.15.53:0/901023294 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2ba080 0x55f58c2beb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5cd9ffb640 1 -- 172.21.15.53:0/2175807025 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f5cd40f1cd0 con 0x7f5cdc1086d0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5ce1665640 1 -- 172.21.15.53:0/901023294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 msgr2=0x55f58c2b9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5ce1665640 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5ce1665640 1 -- 172.21.15.53:0/901023294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2ba080 msgr2=0x55f58c2beb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5ce1665640 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2ba080 0x55f58c2beb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5ce1665640 1 -- 172.21.15.53:0/901023294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f58c1d6750 con 0x55f58c2bf1f0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5ccc0a3cc0 0x7f5ccc0a6180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc19a010 tx=0x7f5cc80138f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5ce1665640 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c1680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40098d0 tx=0x7f5cd402f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70c623f7fde51ee server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5cc6ffd640 1 -- 172.21.15.53:0/901023294 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd40bdbc0 con 0x55f58c2bf1f0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5cc6ffd640 1 -- 172.21.15.53:0/901023294 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cd4036070 con 0x55f58c2bf1f0 2024-04-28T12:50:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.311+0000 7f5cc6ffd640 1 -- 172.21.15.53:0/901023294 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd403bea0 con 0x55f58c2bf1f0 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- 172.21.15.53:0/901023294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2bf1f0 msgr2=0x55f58c2c1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c1680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40098d0 tx=0x7f5cd402f840 comp rx=0 tx=0).stop 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- 172.21.15.53:0/901023294 shutdown_connections 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c1680 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2ba080 0x55f58c2beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/901023294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- 172.21.15.53:0/901023294 >> 172.21.15.53:0/901023294 conn(0x55f58c2b4d00 msgr2=0x55f58c2b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- 172.21.15.53:0/901023294 shutdown_connections 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- 172.21.15.53:0/901023294 wait complete. 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- start start 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 Processor -- start 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2b9690 0x55f58c2c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2bf1f0 0x55f58c2cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f58c1ba3a0 con 0x55f58c2ba080 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f58c1a5fe0 con 0x55f58c2b9690 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f58c21e740 con 0x55f58c2bf1f0 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.312+0000 7f5ce5de3740 10 client.0 _ll_register_callbacks cb 0x55f58c1bc150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2c3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36160/0 (socket says 172.21.15.53:36160) 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 -- 172.21.15.53:0/1267173283 learned_addr learned my addr 172.21.15.53:0/1267173283 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2b9690 0x55f58c2c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5ce1665640 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2bf1f0 0x55f58c2cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 -- 172.21.15.53:0/1267173283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2bf1f0 msgr2=0x55f58c2cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2bf1f0 0x55f58c2cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 -- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2b9690 msgr2=0x55f58c2c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2b9690 0x55f58c2c3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 -- 172.21.15.53:0/1267173283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f58bf302d0 con 0x55f58c2ba080 2024-04-28T12:50:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2c3b80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8003850 tx=0x7f5cc8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789d2052c4f794f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cbbfff640 1 -- 172.21.15.53:0/1267173283 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8023030 con 0x55f58c2ba080 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cbbfff640 1 -- 172.21.15.53:0/1267173283 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f5cc8003d10 con 0x55f58c2ba080 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cd4036070 con 0x55f58c2ba080 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5cbbfff640 1 -- 172.21.15.53:0/1267173283 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5cc802f050 con 0x55f58c2ba080 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5ce5de3740 10 client.15072 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.313+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f58bffa6b0 con 0x55f58c2ba080 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5cbbfff640 1 -- 172.21.15.53:0/1267173283 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cd4036070 con 0x55f58c2ba080 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5cbbfff640 1 -- 172.21.15.53:0/1267173283 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f5cc801fa90 con 0x55f58c2ba080 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5ce5de3740 10 client.15072 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5ce5de3740 10 client.15072 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5ce5de3740 20 client.15072 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c2d0190 0x55f58c2d1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5ce5de3740 4 client.15072 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 --> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f58c22c350 con 0x55f58c2d0190 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.314+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c2d0190 0x55f58c2d1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.315+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c2d0190 0x55f58c2d1600 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aca13002e22c2822 server_cookie=c10d5fc525e2d475 in_seq=0 out_seq=0 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.315+0000 7f5cbbfff640 10 client.15072 ms_handle_connect on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.315+0000 7f5cbbfff640 1 -- 172.21.15.53:0/1267173283 <== mds.1 v2:172.21.15.77:6836/2147382100 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f58c22c350 con 0x55f58c2d0190 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.315+0000 7f5cbbfff640 10 client.15072 handle_command_reply: tid=0 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.315+0000 7f5ce5de3740 1 client.15072 shutdown 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.315+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 msgr2=0x55f58c2c3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.315+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2c3b80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8003850 tx=0x7f5cc8019000 comp rx=0 tx=0).stop 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 shutdown_connections 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2bf1f0 0x55f58c2cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c2d0190 0x55f58c2d1600 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2c3b80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/1267173283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2b9690 0x55f58c2c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5cbbfff640 0 client.15072 ms_handle_reset on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:11.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 >> 172.21.15.53:0/1267173283 conn(0x55f58c2b4d00 msgr2=0x55f58c2b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 shutdown_connections 2024-04-28T12:50:11.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 1 -- 172.21.15.53:0/1267173283 wait complete. 2024-04-28T12:50:11.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.316+0000 7f5ce5de3740 20 client.15072 trim_cache size 0 max 16384 2024-04-28T12:50:11.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 Processor -- start 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 -- start start 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2cab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f58c2cf940 con 0x55f58c2ba080 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f58c2d0010 con 0x55f58c2bf1f0 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f58c2d06e0 con 0x55f58c2b9690 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36164/0 (socket says 172.21.15.53:36164) 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 -- 172.21.15.53:0/476955688 learned_addr learned my addr 172.21.15.53:0/476955688 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 -- 172.21.15.53:0/476955688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 msgr2=0x55f58c2b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 -- 172.21.15.53:0/476955688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 msgr2=0x55f58c2c3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5cdbfff640 1 -- 172.21.15.53:0/476955688 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f58c2cf770 con 0x55f58c2ba080 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.334+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2cab80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8003820 tx=0x7f5cc800b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6965fa9226b4d88f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5cc67fc640 1 -- 172.21.15.53:0/476955688 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8038070 con 0x55f58c2ba080 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5cc67fc640 1 -- 172.21.15.53:0/476955688 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cc8011e00 con 0x55f58c2ba080 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5cc67fc640 1 -- 172.21.15.53:0/476955688 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8023040 con 0x55f58c2ba080 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 -- 172.21.15.53:0/476955688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 msgr2=0x55f58c2cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2cab80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8003820 tx=0x7f5cc800b2e0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 -- 172.21.15.53:0/476955688 shutdown_connections 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2ba080 0x55f58c2cab80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/476955688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2b9690 0x55f58c2b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 -- 172.21.15.53:0/476955688 >> 172.21.15.53:0/476955688 conn(0x55f58c2b4d00 msgr2=0x55f58c2bb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 -- 172.21.15.53:0/476955688 shutdown_connections 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.335+0000 7f5ce5de3740 1 -- 172.21.15.53:0/476955688 wait complete. 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 -- start start 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 Processor -- start 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2b9690 0x55f58c2c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2ba080 0x55f58c2c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f58c2d1480 con 0x55f58c2b9690 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f58c2d0db0 con 0x55f58c2bf1f0 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f58c2d1b50 con 0x55f58c2ba080 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce5de3740 10 client.0 _ll_register_callbacks cb 0x7f5cc800baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2b9690 0x55f58c2c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2b9690 0x55f58c2c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36170/0 (socket says 172.21.15.53:36170) 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 -- 172.21.15.53:0/3849972402 learned_addr learned my addr 172.21.15.53:0/3849972402 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce1665640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2ba080 0x55f58c2c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 -- 172.21.15.53:0/3849972402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2ba080 msgr2=0x55f58c2c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2ba080 0x55f58c2c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 -- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 msgr2=0x55f58c2c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5ce0e64640 1 -- 172.21.15.53:0/3849972402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f58c2ca470 con 0x55f58c2b9690 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.336+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2ba080 0x55f58c2c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce0e64640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2b9690 0x55f58c2c6920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0004700 tx=0x7f5cd0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8129d0e7bc33fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5cbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5cbbfff640 1 -- 172.21.15.53:0/3849972402 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd001b390 con 0x55f58c2b9690 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5cbbfff640 1 -- 172.21.15.53:0/3849972402 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f5cd001b530 con 0x55f58c2b9690 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cc8011e00 con 0x55f58c2b9690 2024-04-28T12:50:11.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5cbbfff640 1 -- 172.21.15.53:0/3849972402 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5cd001b730 con 0x55f58c2b9690 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 10 client.15078 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f58c336800 con 0x55f58c2b9690 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5cbbfff640 1 -- 172.21.15.53:0/3849972402 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cc8011e00 con 0x55f58c2b9690 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5cbbfff640 1 -- 172.21.15.53:0/3849972402 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f5cd0004a90 con 0x55f58c2b9690 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 10 client.15078 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 10 client.15078 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 20 client.15078 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c33b460 0x55f58c33a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 4 client.15078 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.337+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 --> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] -- command(tid 0: {"prefix": "status"}) -- 0x55f58c2261c0 con 0x55f58c33b460 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c33b460 0x55f58c33a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5cdbfff640 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c33b460 0x55f58c33a170 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb9867eab73e188c server_cookie=6bf6f7df963cbc18 in_seq=0 out_seq=0 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5cbbfff640 10 client.15078 ms_handle_connect on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5cbbfff640 1 -- 172.21.15.53:0/3849972402 <== mds.1 v2:172.21.15.77:6836/2147382100 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55f58c2261c0 con 0x55f58c33b460 2024-04-28T12:50:11.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5cbbfff640 10 client.15078 handle_command_reply: tid=0 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 client.15078 shutdown 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2b9690 msgr2=0x55f58c2c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2b9690 0x55f58c2c6920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0004700 tx=0x7f5cd0011b00 comp rx=0 tx=0).stop 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 shutdown_connections 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f58c2bf1f0 0x55f58c2c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f58c2ba080 0x55f58c2c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55f58c33b460 0x55f58c33a170 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 --2- 172.21.15.53:0/3849972402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f58c2b9690 0x55f58c2c6920 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5cbbfff640 0 client.15078 ms_handle_reset on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.338+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 >> 172.21.15.53:0/3849972402 conn(0x55f58c2b4d00 msgr2=0x55f58c2c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 shutdown_connections 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce5de3740 1 -- 172.21.15.53:0/3849972402 wait complete. 2024-04-28T12:50:11.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce5de3740 20 client.15078 trim_cache size 0 max 16384 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 1, 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24485, 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 83.805513155, 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 22, 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 88, 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 86, 2024-04-28T12:50:11.342 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 104.223321494 2024-04-28T12:50:11.343 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-28T12:50:11.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5ccc0a3cc0 msgr2=0x7f5ccc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5ccc0a3cc0 0x7f5ccc0a6180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc19a010 tx=0x7f5cc80138f0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 msgr2=0x7f5cdc1a00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc1a00b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097b0 tx=0x7f5cd4003c20 comp rx=0 tx=0).stop 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 shutdown_connections 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5ccc0a3cc0 0x7f5ccc0a6180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cdc1086d0 0x7f5cdc1a00b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cdc105da0 0x7f5cdc199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 --2- 172.21.15.53:0/2175807025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cdc103470 0x7f5cdc198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 >> 172.21.15.53:0/2175807025 conn(0x7f5cdc0fce20 msgr2=0x7f5cdc103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 shutdown_connections 2024-04-28T12:50:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.339+0000 7f5ce30c8640 1 -- 172.21.15.53:0/2175807025 wait complete. 2024-04-28T12:50:11.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-28T12:50:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 -- 172.21.15.53:0/3548920530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0105d40 msgr2=0x7f9ad0108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 --2- 172.21.15.53:0/3548920530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0105d40 0x7f9ad0108130 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009900 tx=0x7f9ac402a410 comp rx=0 tx=0).stop 2024-04-28T12:50:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 -- 172.21.15.53:0/3548920530 shutdown_connections 2024-04-28T12:50:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 --2- 172.21.15.53:0/3548920530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad0108670 0x7f9ad010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 --2- 172.21.15.53:0/3548920530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0105d40 0x7f9ad0108130 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 --2- 172.21.15.53:0/3548920530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ad0103410 0x7f9ad0105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 -- 172.21.15.53:0/3548920530 >> 172.21.15.53:0/3548920530 conn(0x7f9ad00fce20 msgr2=0x7f9ad00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 -- 172.21.15.53:0/3548920530 shutdown_connections 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 -- 172.21.15.53:0/3548920530 wait complete. 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 Processor -- start 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.453+0000 7f9ad7740640 1 -- start start 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ad0103410 0x7f9ad019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad0105d40 0x7f9ad019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0108670 0x7f9ad01a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad010e4d0 con 0x7f9ad0108670 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad010e350 con 0x7f9ad0105d40 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad010e650 con 0x7f9ad0103410 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0108670 0x7f9ad01a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0108670 0x7f9ad01a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36198/0 (socket says 172.21.15.53:36198) 2024-04-28T12:50:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 -- 172.21.15.53:0/224820278 learned_addr learned my addr 172.21.15.53:0/224820278 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad0105d40 0x7f9ad019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ad0103410 0x7f9ad019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 -- 172.21.15.53:0/224820278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ad0103410 msgr2=0x7f9ad019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ad0103410 0x7f9ad019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 -- 172.21.15.53:0/224820278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad0105d40 msgr2=0x7f9ad019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad0105d40 0x7f9ad019b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 -- 172.21.15.53:0/224820278 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad01a29f0 con 0x7f9ad0108670 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ad0103410 0x7f9ad019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad5cdd640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0108670 0x7f9ad01a2380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c970 tx=0x7f9ac800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ace7fc640 1 -- 172.21.15.53:0/224820278 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8007bb0 con 0x7f9ad0108670 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad01a2ce0 con 0x7f9ad0108670 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ace7fc640 1 -- 172.21.15.53:0/224820278 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ac8007d50 con 0x7f9ad0108670 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ace7fc640 1 -- 172.21.15.53:0/224820278 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8018770 con 0x7f9ad0108670 2024-04-28T12:50:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.454+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad01a31b0 con 0x7f9ad0108670 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ace7fc640 1 -- 172.21.15.53:0/224820278 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9ac800f990 con 0x7f9ad0108670 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 Processor -- start 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 -- start start 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e59ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 0x561c47e5eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5f1f0 0x561c47e61680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561c47dbe740 con 0x561c47e5f1f0 2024-04-28T12:50:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c47d5a3a0 con 0x561c47e59690 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ada45c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x561c47dba700 con 0x561c47e5a080 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.457+0000 7f9ace7fc640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ab80a3c40 0x7f9ab80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5f1f0 0x561c47e61680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5f1f0 0x561c47e61680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36214/0 (socket says 172.21.15.53:36214) 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/2733025037 learned_addr learned my addr 172.21.15.53:0/2733025037 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad5cdd640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ab80a3c40 0x7f9ab80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e59ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 0x561c47e5eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ace7fc640 1 -- 172.21.15.53:0/224820278 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f9ac80d06d0 con 0x7f9ad0108670 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/2733025037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 msgr2=0x561c47e5eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 0x561c47e5eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/2733025037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 msgr2=0x561c47e59ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e59ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/2733025037 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c47d76750 con 0x561c47e5f1f0 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad5cdd640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ab80a3c40 0x7f9ab80a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9ac80a4f90 tx=0x7f9ac8097270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5f1f0 0x561c47e61680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4005940 tx=0x7f9ac40058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7f7b5e4cfdd569 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ab77fe640 1 -- 172.21.15.53:0/2733025037 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4035050 con 0x561c47e5f1f0 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ab77fe640 1 -- 172.21.15.53:0/2733025037 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ac4004230 con 0x561c47e5f1f0 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.458+0000 7f9ab77fe640 1 -- 172.21.15.53:0/2733025037 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac403d450 con 0x561c47e5f1f0 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 -- 172.21.15.53:0/2733025037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5f1f0 msgr2=0x561c47e61680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5f1f0 0x561c47e61680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4005940 tx=0x7f9ac40058b0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 -- 172.21.15.53:0/2733025037 shutdown_connections 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5f1f0 0x561c47e61680 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 0x561c47e5eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 --2- 172.21.15.53:0/2733025037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e59ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 -- 172.21.15.53:0/2733025037 >> 172.21.15.53:0/2733025037 conn(0x561c47e54d00 msgr2=0x561c47e55100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 -- 172.21.15.53:0/2733025037 shutdown_connections 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 -- 172.21.15.53:0/2733025037 wait complete. 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.459+0000 7f9ada45c740 1 -- start start 2024-04-28T12:50:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 1 Processor -- start 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e63ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5a080 0x561c47e64020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 0x561c47e6b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561c47e62140 con 0x561c47e59690 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c47d45fe0 con 0x561c47e5a080 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x561c47e622c0 con 0x561c47e5f1f0 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ada45c740 10 client.0 _ll_register_callbacks cb 0x561c47d5c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e63ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e63ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36226/0 (socket says 172.21.15.53:36226) 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/972286042 learned_addr learned my addr 172.21.15.53:0/972286042 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 0x561c47e6b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5a080 0x561c47e64020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 msgr2=0x561c47e6b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 0x561c47e6b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/972286042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5a080 msgr2=0x561c47e64020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5a080 0x561c47e64020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/972286042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c47ad02d0 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5a080 0x561c47e64020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e63ae0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4000c00 tx=0x7f9ac40043e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36af5c850b0988d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.460+0000 7f9ab4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/972286042 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4035070 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/972286042 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f9ac403d450 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ac4004230 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/972286042 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9ac40044d0 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 10 client.15090 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561c47b9a6b0 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/972286042 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ac4004230 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/972286042 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f9ac404d670 con 0x561c47e59690 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 10 client.15090 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:11.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 10 client.15090 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 20 client.15090 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47e70000 0x561c47e71470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 4 client.15090 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.461+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 --> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c47dcc350 con 0x561c47e70000 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.462+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47e70000 0x561c47e71470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.462+0000 7f9ab4ff9640 10 client.15090 ms_handle_connect on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.462+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47e70000 0x561c47e71470 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a5f090c211c1b0 server_cookie=1378e10ffc1c9440 in_seq=0 out_seq=0 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/972286042 <== mds.0 v2:172.21.15.191:6838/2399731075 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561c47dcc350 con 0x561c47e70000 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ab4ff9640 10 client.15090 handle_command_reply: tid=0 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 client.15090 shutdown 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 msgr2=0x561c47e63ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e63ae0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4000c00 tx=0x7f9ac40043e0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 shutdown_connections 2024-04-28T12:50:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47e70000 0x561c47e71470 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 0x561c47e6b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ab4ff9640 0 client.15090 ms_handle_reset on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5a080 0x561c47e64020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 --2- 172.21.15.53:0/972286042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e63ae0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 >> 172.21.15.53:0/972286042 conn(0x561c47e54d00 msgr2=0x561c47e53f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 shutdown_connections 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 1 -- 172.21.15.53:0/972286042 wait complete. 2024-04-28T12:50:11.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.463+0000 7f9ada45c740 20 client.15090 trim_cache size 0 max 16384 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 Processor -- start 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 -- start start 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e626c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5a080 0x561c47e6aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 0x561c47e636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561c47e6f7b0 con 0x561c47e5a080 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c47e6fe80 con 0x561c47e59690 2024-04-28T12:50:11.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ada45c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x561c47e70550 con 0x561c47e5f1f0 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5a080 0x561c47e6aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5a080 0x561c47e6aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36230/0 (socket says 172.21.15.53:36230) 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 -- 172.21.15.53:0/495415276 learned_addr learned my addr 172.21.15.53:0/495415276 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e626c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 -- 172.21.15.53:0/495415276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 msgr2=0x561c47e636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 0x561c47e636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 -- 172.21.15.53:0/495415276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 msgr2=0x561c47e626c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e626c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 -- 172.21.15.53:0/495415276 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c47e55a70 con 0x561c47e5a080 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5a080 0x561c47e6aff0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x561c47e5aaf0 tx=0x7f9ac00086f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e9f79d5d58e818 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ab6ffd640 1 -- 172.21.15.53:0/495415276 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0019330 con 0x561c47e5a080 2024-04-28T12:50:11.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ab6ffd640 1 -- 172.21.15.53:0/495415276 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ac001d030 con 0x561c47e5a080 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.482+0000 7f9ab6ffd640 1 -- 172.21.15.53:0/495415276 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac000fe40 con 0x561c47e5a080 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 -- 172.21.15.53:0/495415276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5a080 msgr2=0x561c47e6aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5a080 0x561c47e6aff0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x561c47e5aaf0 tx=0x7f9ac00086f0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 -- 172.21.15.53:0/495415276 shutdown_connections 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5f1f0 0x561c47e636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e5a080 0x561c47e6aff0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 --2- 172.21.15.53:0/495415276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e59690 0x561c47e626c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 -- 172.21.15.53:0/495415276 >> 172.21.15.53:0/495415276 conn(0x561c47e54d00 msgr2=0x561c47e550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 -- 172.21.15.53:0/495415276 shutdown_connections 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 -- 172.21.15.53:0/495415276 wait complete. 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 -- start start 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.483+0000 7f9ada45c740 1 Processor -- start 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e66b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 0x561c47e670b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ada45c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5f1f0 0x561c47e6a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ada45c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x561c47e712f0 con 0x561c47e59690 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ada45c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c47e70c20 con 0x561c47e5f1f0 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ada45c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x561c47e719c0 con 0x561c47e5a080 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ada45c740 10 client.0 _ll_register_callbacks cb 0x7f9ac4032ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e66b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e66b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36244/0 (socket says 172.21.15.53:36244) 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/3573083809 learned_addr learned my addr 172.21.15.53:0/3573083809 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/3573083809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 msgr2=0x561c47e670b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 0x561c47e670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/3573083809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5f1f0 msgr2=0x561c47e6a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5f1f0 0x561c47e6a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 -- 172.21.15.53:0/3573083809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c47e6a6c0 con 0x561c47e59690 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ad4cdb640 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e66b70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4033000 tx=0x7f9ac40319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993c2f1a1782543d server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:11.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ab4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.484+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/3573083809 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4005d50 con 0x561c47e59690 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ac001d030 con 0x561c47e59690 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/3573083809 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f9ac4003a00 con 0x561c47e59690 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/3573083809 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9ac404d440 con 0x561c47e59690 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 10 client.15096 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561c47ed6490 con 0x561c47e59690 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/3573083809 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ac001d030 con 0x561c47e59690 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/3573083809 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f9ac406a030 con 0x561c47e59690 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 10 client.15096 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 10 client.15096 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 20 client.15096 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47edafa0 0x561c47ed73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 4 client.15096 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:11.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 --> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] -- command(tid 0: {"prefix": "status"}) -- 0x561c47dc61c0 con 0x561c47edafa0 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.485+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47edafa0 0x561c47ed73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ab4ff9640 10 client.15096 ms_handle_connect on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ad54dc640 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47edafa0 0x561c47ed73e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8163e74e064016d0 server_cookie=9a2ff6e11d352b63 in_seq=0 out_seq=0 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ab4ff9640 1 -- 172.21.15.53:0/3573083809 <== mds.0 v2:172.21.15.191:6838/2399731075 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561c47dc61c0 con 0x561c47edafa0 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ab4ff9640 10 client.15096 handle_command_reply: tid=0 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 client.15096 shutdown 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 msgr2=0x561c47e66b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e66b70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4033000 tx=0x7f9ac40319b0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 shutdown_connections 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x561c47edafa0 0x561c47ed73e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c47e5f1f0 0x561c47e6a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561c47e5a080 0x561c47e670b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 --2- 172.21.15.53:0/3573083809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561c47e59690 0x561c47e66b70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ab4ff9640 0 client.15096 ms_handle_reset on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.486+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 >> 172.21.15.53:0/3573083809 conn(0x561c47e54d00 msgr2=0x561c47e63560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 shutdown_connections 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ada45c740 1 -- 172.21.15.53:0/3573083809 wait complete. 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ada45c740 20 client.15096 trim_cache size 0 max 16384 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24505, 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-04-28T12:50:11.489 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-04-28T12:50:11.490 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 86.972372570000005, 2024-04-28T12:50:11.490 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 22, 2024-04-28T12:50:11.490 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 88, 2024-04-28T12:50:11.490 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 86, 2024-04-28T12:50:11.490 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 101.465706756 2024-04-28T12:50:11.490 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ab80a3c40 msgr2=0x7f9ab80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ab80a3c40 0x7f9ab80a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9ac80a4f90 tx=0x7f9ac8097270 comp rx=0 tx=0).stop 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0108670 msgr2=0x7f9ad01a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0108670 0x7f9ad01a2380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c970 tx=0x7f9ac800ce30 comp rx=0 tx=0).stop 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 shutdown_connections 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ab80a3c40 0x7f9ab80a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ad0108670 0x7f9ad01a2380 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ad0105d40 0x7f9ad019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 --2- 172.21.15.53:0/224820278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ad0103410 0x7f9ad019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 >> 172.21.15.53:0/224820278 conn(0x7f9ad00fce20 msgr2=0x7f9ad0102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 shutdown_connections 2024-04-28T12:50:11.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:11.487+0000 7f9ad7740640 1 -- 172.21.15.53:0/224820278 wait complete. 2024-04-28T12:50:11.497 INFO:tasks.cephadm:Setup complete, yielding 2024-04-28T12:50:11.498 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-28T12:50:11.508 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-28T12:50:11.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-28T12:50:11.755 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:11.966 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:11 smithi053 ceph-mon[29182]: pgmap v336: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:11.966 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:11 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/740462285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:11 smithi191 ceph-mon[33016]: pgmap v336: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:12.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:11 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/740462285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:12.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:11 smithi077 ceph-mon[32553]: pgmap v336: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:12.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:11 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/740462285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:13.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.453+0000 7f0f461ee700 1 -- 172.21.15.53:0/790598610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 msgr2=0x7f0f40103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.453+0000 7f0f461ee700 1 --2- 172.21.15.53:0/790598610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40103a20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0f340099c0 tx=0x7f0f34009cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 -- 172.21.15.53:0/790598610 shutdown_connections 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 --2- 172.21.15.53:0/790598610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f40103f60 0x7f0f4010a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 --2- 172.21.15.53:0/790598610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f401023c0 0x7f0f401027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 --2- 172.21.15.53:0/790598610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40103a20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 -- 172.21.15.53:0/790598610 >> 172.21.15.53:0/790598610 conn(0x7f0f400fd980 msgr2=0x7f0f400ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 -- 172.21.15.53:0/790598610 shutdown_connections 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 -- 172.21.15.53:0/790598610 wait complete. 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 Processor -- start 2024-04-28T12:50:13.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.454+0000 7f0f461ee700 1 -- start start 2024-04-28T12:50:13.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f461ee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f401023c0 0x7f0f40199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:13.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f461ee700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:13.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f461ee700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f40103f60 0x7f0f401a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:13.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f461ee700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4010de30 con 0x7f0f401035c0 2024-04-28T12:50:13.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f461ee700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4010dcb0 con 0x7f0f401023c0 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f461ee700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4010dfb0 con 0x7f0f40103f60 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36258/0 (socket says 172.21.15.53:36258) 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 -- 172.21.15.53:0/2448370969 learned_addr learned my addr 172.21.15.53:0/2448370969 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3f7fe700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f401023c0 0x7f0f40199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3ffff700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f40103f60 0x7f0f401a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 -- 172.21.15.53:0/2448370969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f40103f60 msgr2=0x7f0f401a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f40103f60 0x7f0f401a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 -- 172.21.15.53:0/2448370969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f401023c0 msgr2=0x7f0f40199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f401023c0 0x7f0f40199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 -- 172.21.15.53:0/2448370969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f401a11f0 con 0x7f0f401035c0 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3ffff700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f40103f60 0x7f0f401a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3effd700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40199ad0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0f34005910 tx=0x7f0f34005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:13.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3cff9700 1 -- 172.21.15.53:0/2448370969 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f34027070 con 0x7f0f401035c0 2024-04-28T12:50:13.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3cff9700 1 -- 172.21.15.53:0/2448370969 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f340040a0 con 0x7f0f401035c0 2024-04-28T12:50:13.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f401a1480 con 0x7f0f401035c0 2024-04-28T12:50:13.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.455+0000 7f0f3cff9700 1 -- 172.21.15.53:0/2448370969 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f34022ce0 con 0x7f0f401035c0 2024-04-28T12:50:13.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.456+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f401a1960 con 0x7f0f401035c0 2024-04-28T12:50:13.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.456+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f4006a110 con 0x7f0f401035c0 2024-04-28T12:50:13.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.458+0000 7f0f3cff9700 1 -- 172.21.15.53:0/2448370969 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0f3402a070 con 0x7f0f401035c0 2024-04-28T12:50:13.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.458+0000 7f0f3cff9700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0f280a3cd0 0x7f0f280a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:13.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.458+0000 7f0f3f7fe700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0f280a3cd0 0x7f0f280a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:13.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.458+0000 7f0f3cff9700 1 -- 172.21.15.53:0/2448370969 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f0f340d1230 con 0x7f0f401035c0 2024-04-28T12:50:13.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.459+0000 7f0f3f7fe700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0f280a3cd0 0x7f0f280a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0f40103420 tx=0x7f0f3000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:13.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.463+0000 7f0f3cff9700 1 -- 172.21.15.53:0/2448370969 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0f34013070 con 0x7f0f401035c0 2024-04-28T12:50:13.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.688+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f0f400ffb60 con 0x7f0f280a3cd0 2024-04-28T12:50:13.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.689+0000 7f0f3cff9700 1 -- 172.21.15.53:0/2448370969 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f0f400ffb60 con 0x7f0f280a3cd0 2024-04-28T12:50:13.685 INFO:teuthology.orchestra.run.smithi053.stdout:Backend: cephadm 2024-04-28T12:50:13.686 INFO:teuthology.orchestra.run.smithi053.stdout:Available: Yes 2024-04-28T12:50:13.686 INFO:teuthology.orchestra.run.smithi053.stdout:Paused: No 2024-04-28T12:50:13.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0f280a3cd0 msgr2=0x7f0f280a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:13.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0f280a3cd0 0x7f0f280a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0f40103420 tx=0x7f0f3000b410 comp rx=0 tx=0).stop 2024-04-28T12:50:13.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 msgr2=0x7f0f40199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:13.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40199ad0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0f34005910 tx=0x7f0f34005b00 comp rx=0 tx=0).stop 2024-04-28T12:50:13.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 shutdown_connections 2024-04-28T12:50:13.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0f280a3cd0 0x7f0f280a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f40103f60 0x7f0f401a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f401023c0 0x7f0f40199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 --2- 172.21.15.53:0/2448370969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f401035c0 0x7f0f40199ad0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:13.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 >> 172.21.15.53:0/2448370969 conn(0x7f0f400fd980 msgr2=0x7f0f400ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:13.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 shutdown_connections 2024-04-28T12:50:13.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:13.692+0000 7f0f461ee700 1 -- 172.21.15.53:0/2448370969 wait complete. 2024-04-28T12:50:13.994 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:13 smithi053 ceph-mon[29182]: pgmap v337: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:14.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:13 smithi191 ceph-mon[33016]: pgmap v337: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:14.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:13 smithi077 ceph-mon[32553]: pgmap v337: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:14.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-28T12:50:14.676 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:15.032 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:14 smithi053 ceph-mon[29182]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:15.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:14 smithi191 ceph-mon[33016]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:15.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:14 smithi077 ceph-mon[32553]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:15.863 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:15 smithi053 ceph-mon[29182]: pgmap v338: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:16.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:15 smithi191 ceph-mon[33016]: pgmap v338: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:16.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:15 smithi077 ceph-mon[32553]: pgmap v338: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 -- 172.21.15.53:0/1989449758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541046a0 msgr2=0x7f6854104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 --2- 172.21.15.53:0/1989449758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541046a0 0x7f6854104b00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f68440099c0 tx=0x7f6844009cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 -- 172.21.15.53:0/1989449758 shutdown_connections 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 --2- 172.21.15.53:0/1989449758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6854105040 0x7f685410b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 --2- 172.21.15.53:0/1989449758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68541034a0 0x7f68541038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 --2- 172.21.15.53:0/1989449758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541046a0 0x7f6854104b00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 -- 172.21.15.53:0/1989449758 >> 172.21.15.53:0/1989449758 conn(0x7f68540fea40 msgr2=0x7f6854100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 -- 172.21.15.53:0/1989449758 shutdown_connections 2024-04-28T12:50:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.391+0000 7f68590be700 1 -- 172.21.15.53:0/1989449758 wait complete. 2024-04-28T12:50:16.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 Processor -- start 2024-04-28T12:50:16.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 -- start start 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541034a0 0x7f68541995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68541046a0 0x7f6854199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6854105040 0x7f68541a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f685410ef40 con 0x7f68541034a0 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f685410edc0 con 0x7f68541046a0 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.392+0000 7f68590be700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f685410f0c0 con 0x7f6854105040 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541034a0 0x7f68541995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541034a0 0x7f68541995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36276/0 (socket says 172.21.15.53:36276) 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 -- 172.21.15.53:0/204405285 learned_addr learned my addr 172.21.15.53:0/204405285 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f685259c700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68541046a0 0x7f6854199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f685359e700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6854105040 0x7f68541a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 -- 172.21.15.53:0/204405285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6854105040 msgr2=0x7f68541a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6854105040 0x7f68541a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 -- 172.21.15.53:0/204405285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68541046a0 msgr2=0x7f6854199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68541046a0 0x7f6854199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 -- 172.21.15.53:0/204405285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6854075650 con 0x7f68541034a0 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f685359e700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6854105040 0x7f68541a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f6852d9d700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541034a0 0x7f68541995b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f683c00e980 tx=0x7f683c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f684bfff700 1 -- 172.21.15.53:0/204405285 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c00cbd0 con 0x7f68541034a0 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f684bfff700 1 -- 172.21.15.53:0/204405285 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f683c0044e0 con 0x7f68541034a0 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6854075940 con 0x7f68541034a0 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f684bfff700 1 -- 172.21.15.53:0/204405285 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c003e00 con 0x7f68541034a0 2024-04-28T12:50:16.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.393+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6854075e30 con 0x7f68541034a0 2024-04-28T12:50:16.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.395+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f685406a110 con 0x7f68541034a0 2024-04-28T12:50:16.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.396+0000 7f684bfff700 1 -- 172.21.15.53:0/204405285 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f683c003fa0 con 0x7f68541034a0 2024-04-28T12:50:16.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.397+0000 7f684bfff700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68400a3cd0 0x7f68400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.397+0000 7f685259c700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68400a3cd0 0x7f68400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.397+0000 7f684bfff700 1 -- 172.21.15.53:0/204405285 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f683c0d0a40 con 0x7f68541034a0 2024-04-28T12:50:16.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.397+0000 7f685259c700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68400a3cd0 0x7f68400a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6844000c00 tx=0x7f6844005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:16.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.400+0000 7f684bfff700 1 -- 172.21.15.53:0/204405285 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f683c09d280 con 0x7f68541034a0 2024-04-28T12:50:16.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.620+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f685410a4f0 con 0x7f68400a3cd0 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.634+0000 7f684bfff700 1 -- 172.21.15.53:0/204405285 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f685410a4f0 con 0x7f68400a3cd0 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stdout:mds.a smithi053 running (101s) 15s ago 100s 16.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9bc0ae78e386 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stdout:mds.b smithi077 running (98s) 32s ago 98s 13.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 5a6a25b915fb 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stdout:mds.c smithi191 running (113s) 32s ago 112s 19.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 3cb94f0816f6 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stdout:mds.d smithi053 running (2m) 15s ago 2m 16.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 97fb01b2bc92 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stdout:mds.e smithi077 running (118s) 32s ago 118s 15.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 e1f58deb4546 2024-04-28T12:50:16.631 INFO:teuthology.orchestra.run.smithi053.stdout:mds.f smithi191 running (115s) 32s ago 115s 16.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 01f68a3a61a2 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mds.g smithi053 running (104s) 15s ago 103s 16.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 ce9d304d4420 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mds.h smithi077 running (109s) 32s ago 109s 16.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 b8b012781aba 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mds.i smithi191 running (107s) 32s ago 106s 25.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f60ee5b15a98 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mds.j smithi053 running (2m) 15s ago 2m 15.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 534b2f156e3a 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mds.k smithi077 running (2m) 32s ago 2m 19.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 231e8a50d16b 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mds.l smithi191 running (2m) 32s ago 2m 15.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 6196d3c0dcf5 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.x smithi053 *:9283,8765 running (9m) 15s ago 9m 514M - 19.0.0-3306-gbc612ffb 55dc54019ad8 271ec44b4c36 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.y smithi077 *:8443,8765 running (7m) 32s ago 7m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 371a85653376 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.z smithi191 *:8443,8765 running (7m) 32s ago 7m 453M - 19.0.0-3306-gbc612ffb 55dc54019ad8 0f9a78af91f4 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mon.a smithi053 running (9m) 15s ago 9m 49.1M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 bd4ec601fb04 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mon.b smithi077 running (7m) 32s ago 7m 39.5M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 49b5bb26e089 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:mon.c smithi191 running (7m) 32s ago 7m 39.0M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 1490b39f630f 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (7m) 15s ago 7m 58.9M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 804262456f6b 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi077 running (6m) 32s ago 6m 45.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 d604ee695f4d 2024-04-28T12:50:16.632 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi191 running (6m) 32s ago 6m 46.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 c9e2c686910d 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (5m) 15s ago 5m 46.0M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 a490eade1186 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi077 running (5m) 32s ago 5m 53.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 9463589fc6cd 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi191 running (5m) 32s ago 5m 54.1M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 782073519f7e 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi053 running (4m) 15s ago 4m 51.5M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 44db5a237551 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi077 running (4m) 32s ago 4m 54.0M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 9631fb5cfd18 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.8 smithi191 running (3m) 32s ago 3m 53.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 df4fe2ed32dd 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.9 smithi053 running (3m) 15s ago 3m 51.5M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 67fb2e8494ed 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.10 smithi077 running (2m) 32s ago 2m 52.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 215b1baa3a4f 2024-04-28T12:50:16.633 INFO:teuthology.orchestra.run.smithi053.stdout:osd.11 smithi191 running (2m) 32s ago 2m 41.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 420bd1cea570 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.636+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68400a3cd0 msgr2=0x7f68400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.636+0000 7f68590be700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68400a3cd0 0x7f68400a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6844000c00 tx=0x7f6844005fb0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.636+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541034a0 msgr2=0x7f68541995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.636+0000 7f68590be700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541034a0 0x7f68541995b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f683c00e980 tx=0x7f683c00ec90 comp rx=0 tx=0).stop 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 shutdown_connections 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f68400a3cd0 0x7f68400a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68541034a0 0x7f68541995b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6854105040 0x7f68541a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 --2- 172.21.15.53:0/204405285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68541046a0 0x7f6854199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 >> 172.21.15.53:0/204405285 conn(0x7f68540fea40 msgr2=0x7f6854107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 shutdown_connections 2024-04-28T12:50:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:16.637+0000 7f68590be700 1 -- 172.21.15.53:0/204405285 wait complete. 2024-04-28T12:50:17.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-28T12:50:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:17.893 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:17 smithi053 ceph-mon[29182]: pgmap v339: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:17.894 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:17 smithi053 ceph-mon[29182]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:18.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:17 smithi191 ceph-mon[33016]: pgmap v339: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:18.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:17 smithi191 ceph-mon[33016]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:18.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:17 smithi077 ceph-mon[32553]: pgmap v339: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:18.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:17 smithi077 ceph-mon[32553]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:19.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.229+0000 7f595fefe700 1 -- 172.21.15.53:0/286040806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5958105040 msgr2=0x7f595810b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.229+0000 7f595fefe700 1 --2- 172.21.15.53:0/286040806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5958105040 0x7f595810b8f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f595000b0d0 tx=0x7f595000b3e0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 -- 172.21.15.53:0/286040806 shutdown_connections 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 --2- 172.21.15.53:0/286040806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5958105040 0x7f595810b8f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 --2- 172.21.15.53:0/286040806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59581034a0 0x7f59581038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 --2- 172.21.15.53:0/286040806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59581046a0 0x7f5958104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 -- 172.21.15.53:0/286040806 >> 172.21.15.53:0/286040806 conn(0x7f59580fea40 msgr2=0x7f5958100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 -- 172.21.15.53:0/286040806 shutdown_connections 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 -- 172.21.15.53:0/286040806 wait complete. 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 Processor -- start 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.230+0000 7f595fefe700 1 -- start start 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595fefe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59581034a0 0x7f59581995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595fefe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59581046a0 0x7f5958199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595fefe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5958105040 0x7f59581a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59581034a0 0x7f59581995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59581034a0 0x7f59581995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41642/0 (socket says 172.21.15.53:41642) 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 -- 172.21.15.53:0/1241479826 learned_addr learned my addr 172.21.15.53:0/1241479826 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f595810efc0 con 0x7f59581034a0 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f595810ee40 con 0x7f59581046a0 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f595810f140 con 0x7f5958105040 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595d499700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59581046a0 0x7f5958199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 -- 172.21.15.53:0/1241479826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5958105040 msgr2=0x7f59581a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5958105040 0x7f59581a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 -- 172.21.15.53:0/1241479826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59581046a0 msgr2=0x7f5958199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59581046a0 0x7f5958199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59581a1200 con 0x7f59581034a0 2024-04-28T12:50:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f595dc9a700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59581034a0 0x7f59581995a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f595400b530 tx=0x7f595400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f594effd700 1 -- 172.21.15.53:0/1241479826 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954015020 con 0x7f59581034a0 2024-04-28T12:50:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.231+0000 7f594effd700 1 -- 172.21.15.53:0/1241479826 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5954011ce0 con 0x7f59581034a0 2024-04-28T12:50:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.232+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59581a14f0 con 0x7f59581034a0 2024-04-28T12:50:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.232+0000 7f594effd700 1 -- 172.21.15.53:0/1241479826 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954010d60 con 0x7f59581034a0 2024-04-28T12:50:19.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.232+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59581a1b00 con 0x7f59581034a0 2024-04-28T12:50:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.233+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f595806a110 con 0x7f59581034a0 2024-04-28T12:50:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.234+0000 7f594effd700 1 -- 172.21.15.53:0/1241479826 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5954018020 con 0x7f59581034a0 2024-04-28T12:50:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.234+0000 7f594effd700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f59440a3d50 0x7f59440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.234+0000 7f595d499700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f59440a3d50 0x7f59440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.234+0000 7f594effd700 1 -- 172.21.15.53:0/1241479826 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f59540cfa70 con 0x7f59581034a0 2024-04-28T12:50:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.235+0000 7f595d499700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f59440a3d50 0x7f59440a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5948005950 tx=0x7f59480058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:19.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.239+0000 7f594effd700 1 -- 172.21.15.53:0/1241479826 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f595401f030 con 0x7f59581034a0 2024-04-28T12:50:19.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.459+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f5958063860 con 0x7f59440a3d50 2024-04-28T12:50:19.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.465+0000 7f594effd700 1 -- 172.21.15.53:0/1241479826 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f5958063860 con 0x7f59440a3d50 2024-04-28T12:50:19.462 INFO:teuthology.orchestra.run.smithi053.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-28T12:50:19.462 INFO:teuthology.orchestra.run.smithi053.stdout:mds.all 12/12 35s ago 2m smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi077=b;smithi077=e;smithi077=h;smithi077=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-28T12:50:19.462 INFO:teuthology.orchestra.run.smithi053.stdout:mgr 3/3 35s ago 7m smithi053=x;smithi077=y;smithi191=z;count:3 2024-04-28T12:50:19.462 INFO:teuthology.orchestra.run.smithi053.stdout:mon 3/3 35s ago 8m smithi053:172.21.15.53=a;smithi077:172.21.15.77=b;smithi191:172.21.15.191=c;count:3 2024-04-28T12:50:19.462 INFO:teuthology.orchestra.run.smithi053.stdout:osd 12 35s ago - 2024-04-28T12:50:19.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f59440a3d50 msgr2=0x7f59440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f59440a3d50 0x7f59440a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5948005950 tx=0x7f59480058e0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59581034a0 msgr2=0x7f59581995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59581034a0 0x7f59581995a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f595400b530 tx=0x7f595400b8f0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 shutdown_connections 2024-04-28T12:50:19.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f59440a3d50 0x7f59440a6210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59581034a0 0x7f59581995a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5958105040 0x7f59581a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 --2- 172.21.15.53:0/1241479826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59581046a0 0x7f5958199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 >> 172.21.15.53:0/1241479826 conn(0x7f59580fea40 msgr2=0x7f59581079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:19.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 shutdown_connections 2024-04-28T12:50:19.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:19.467+0000 7f595fefe700 1 -- 172.21.15.53:0/1241479826 wait complete. 2024-04-28T12:50:20.000 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:19 smithi077 ceph-mon[32553]: pgmap v340: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:19 smithi191 ceph-mon[33016]: pgmap v340: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:20.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:19 smithi053 ceph-mon[29182]: pgmap v340: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:20.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-28T12:50:20.355 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:20.778 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:20 smithi053 ceph-mon[29182]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:21.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:20 smithi191 ceph-mon[33016]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:21.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:20 smithi077 ceph-mon[32553]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 -- 172.21.15.53:0/1511514196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 msgr2=0x7f3e5c108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 --2- 172.21.15.53:0/1511514196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c108c50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009990 tx=0x7f3e50009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 -- 172.21.15.53:0/1511514196 shutdown_connections 2024-04-28T12:50:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 --2- 172.21.15.53:0/1511514196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e5c109190 0x7f3e5c10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 --2- 172.21.15.53:0/1511514196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c103f30 0x7f3e5c106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 --2- 172.21.15.53:0/1511514196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c108c50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 -- 172.21.15.53:0/1511514196 >> 172.21.15.53:0/1511514196 conn(0x7f3e5c0fd940 msgr2=0x7f3e5c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:21.980 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:21 smithi053 ceph-mon[29182]: pgmap v341: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 -- 172.21.15.53:0/1511514196 shutdown_connections 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.982+0000 7f3e60831700 1 -- 172.21.15.53:0/1511514196 wait complete. 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 Processor -- start 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 -- start start 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c103f30 0x7f3e5c103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e5c109190 0x7f3e5c102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c10ef60 con 0x7f3e5c106860 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c10ede0 con 0x7f3e5c103f30 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.983+0000 7f3e60831700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c10f0e0 con 0x7f3e5c109190 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e5a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c103f30 0x7f3e5c103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:21.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c101ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41660/0 (socket says 172.21.15.53:41660) 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 -- 172.21.15.53:0/241449566 learned_addr learned my addr 172.21.15.53:0/241449566 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e5ad9d700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e5c109190 0x7f3e5c102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 -- 172.21.15.53:0/241449566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e5c109190 msgr2=0x7f3e5c102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e5c109190 0x7f3e5c102420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 -- 172.21.15.53:0/241449566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c103f30 msgr2=0x7f3e5c103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c103f30 0x7f3e5c103890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 -- 172.21.15.53:0/241449566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c102cd0 con 0x7f3e5c106860 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e59d9b700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c101ee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006010 tx=0x7f3e500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e4b7fe700 1 -- 172.21.15.53:0/241449566 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e50027070 con 0x7f3e5c106860 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5c1a3690 con 0x7f3e5c106860 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e4b7fe700 1 -- 172.21.15.53:0/241449566 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e50022470 con 0x7f3e5c106860 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.984+0000 7f3e4b7fe700 1 -- 172.21.15.53:0/241449566 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e50017650 con 0x7f3e5c106860 2024-04-28T12:50:21.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.985+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5c1a3ae0 con 0x7f3e5c106860 2024-04-28T12:50:21.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.986+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e5c06a110 con 0x7f3e5c106860 2024-04-28T12:50:21.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.987+0000 7f3e4b7fe700 1 -- 172.21.15.53:0/241449566 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3e50009d10 con 0x7f3e5c106860 2024-04-28T12:50:21.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.987+0000 7f3e4b7fe700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3e440a3d50 0x7f3e440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:21.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.988+0000 7f3e5a59c700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3e440a3d50 0x7f3e440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:21.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.988+0000 7f3e4b7fe700 1 -- 172.21.15.53:0/241449566 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f3e500d1050 con 0x7f3e5c106860 2024-04-28T12:50:21.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.988+0000 7f3e5a59c700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3e440a3d50 0x7f3e440a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009c60 tx=0x7f3e4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:21.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:21.992+0000 7f3e4b7fe700 1 -- 172.21.15.53:0/241449566 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3e50013070 con 0x7f3e5c106860 2024-04-28T12:50:22.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:21 smithi191 ceph-mon[33016]: pgmap v341: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:22.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:21 smithi077 ceph-mon[32553]: pgmap v341: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T12:50:22.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.208+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f3e5c063860 con 0x7f3e440a3d50 2024-04-28T12:50:22.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.209+0000 7f3e4b7fe700 1 -- 172.21.15.53:0/241449566 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f3e5c063860 con 0x7f3e440a3d50 2024-04-28T12:50:22.205 INFO:teuthology.orchestra.run.smithi053.stdout:HOST ADDR LABELS STATUS 2024-04-28T12:50:22.206 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 172.21.15.53 2024-04-28T12:50:22.206 INFO:teuthology.orchestra.run.smithi053.stdout:smithi077 172.21.15.77 2024-04-28T12:50:22.206 INFO:teuthology.orchestra.run.smithi053.stdout:smithi191 172.21.15.191 2024-04-28T12:50:22.206 INFO:teuthology.orchestra.run.smithi053.stdout:3 hosts in cluster 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3e440a3d50 msgr2=0x7f3e440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3e440a3d50 0x7f3e440a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009c60 tx=0x7f3e4c009260 comp rx=0 tx=0).stop 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 msgr2=0x7f3e5c101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c101ee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006010 tx=0x7f3e500047b0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 shutdown_connections 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f3e440a3d50 0x7f3e440a6210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e5c109190 0x7f3e5c102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c103f30 0x7f3e5c103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 --2- 172.21.15.53:0/241449566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c106860 0x7f3e5c101ee0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 >> 172.21.15.53:0/241449566 conn(0x7f3e5c0fd940 msgr2=0x7f3e5c104950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 shutdown_connections 2024-04-28T12:50:22.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:22.211+0000 7f3e60831700 1 -- 172.21.15.53:0/241449566 wait complete. 2024-04-28T12:50:22.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-28T12:50:22.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:22 smithi053 ceph-mon[29182]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:23.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:22 smithi191 ceph-mon[33016]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:23.134 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:23.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:22 smithi077 ceph-mon[32553]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:23.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:23 smithi053 ceph-mon[29182]: pgmap v342: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:24.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:23 smithi191 ceph-mon[33016]: pgmap v342: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:24.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:23 smithi077 ceph-mon[32553]: pgmap v342: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 -- 172.21.15.53:0/3362915028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 msgr2=0x7fd5b010a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 --2- 172.21.15.53:0/3362915028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b010a7d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800b0d0 tx=0x7fd5a800b3e0 comp rx=0 tx=0).stop 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 -- 172.21.15.53:0/3362915028 shutdown_connections 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 --2- 172.21.15.53:0/3362915028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b010a7d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 --2- 172.21.15.53:0/3362915028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b0102380 0x7fd5b01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 --2- 172.21.15.53:0/3362915028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5b0103580 0x7fd5b01039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 -- 172.21.15.53:0/3362915028 >> 172.21.15.53:0/3362915028 conn(0x7fd5b00fd920 msgr2=0x7fd5b00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 -- 172.21.15.53:0/3362915028 shutdown_connections 2024-04-28T12:50:24.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.830+0000 7fd5b501b700 1 -- 172.21.15.53:0/3362915028 wait complete. 2024-04-28T12:50:24.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 Processor -- start 2024-04-28T12:50:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 -- start start 2024-04-28T12:50:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b0102380 0x7fd5b01063e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5b0103580 0x7fd5b0104a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b0104f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b010de10 con 0x7fd5b0103f20 2024-04-28T12:50:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b010dc90 con 0x7fd5b0102380 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5b501b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b010df90 con 0x7fd5b0103580 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5af59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b0104f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5af59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b0104f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41688/0 (socket says 172.21.15.53:41688) 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5af59e700 1 -- 172.21.15.53:0/2484153331 learned_addr learned my addr 172.21.15.53:0/2484153331 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5aed9d700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b0102380 0x7fd5b01063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5ae59c700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5b0103580 0x7fd5b0104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.831+0000 7fd5af59e700 1 -- 172.21.15.53:0/2484153331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5b0103580 msgr2=0x7fd5b0104a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5af59e700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5b0103580 0x7fd5b0104a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5af59e700 1 -- 172.21.15.53:0/2484153331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b0102380 msgr2=0x7fd5b01063e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5af59e700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b0102380 0x7fd5b01063e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5af59e700 1 -- 172.21.15.53:0/2484153331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b0105820 con 0x7fd5b0103f20 2024-04-28T12:50:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5ae59c700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5b0103580 0x7fd5b0104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5af59e700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b0104f70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800d010 tx=0x7fd5a8007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd597fff700 1 -- 172.21.15.53:0/2484153331 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8021070 con 0x7fd5b0103f20 2024-04-28T12:50:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd597fff700 1 -- 172.21.15.53:0/2484153331 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5a801db00 con 0x7fd5b0103f20 2024-04-28T12:50:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b01a3690 con 0x7fd5b0103f20 2024-04-28T12:50:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd597fff700 1 -- 172.21.15.53:0/2484153331 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a801bca0 con 0x7fd5b0103f20 2024-04-28T12:50:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.832+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b01a3af0 con 0x7fd5b0103f20 2024-04-28T12:50:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.833+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b006a110 con 0x7fd5b0103f20 2024-04-28T12:50:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.834+0000 7fd597fff700 1 -- 172.21.15.53:0/2484153331 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd5a8024070 con 0x7fd5b0103f20 2024-04-28T12:50:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.835+0000 7fd597fff700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd5980a3cd0 0x7fd5980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.835+0000 7fd5aed9d700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd5980a3cd0 0x7fd5980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.835+0000 7fd597fff700 1 -- 172.21.15.53:0/2484153331 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fd5a800f020 con 0x7fd5b0103f20 2024-04-28T12:50:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.835+0000 7fd5aed9d700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd5980a3cd0 0x7fd5980a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4005fd0 tx=0x7fd5a4005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:24.839+0000 7fd597fff700 1 -- 172.21.15.53:0/2484153331 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd5a802b020 con 0x7fd5b0103f20 2024-04-28T12:50:25.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.055+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fd5b01a3cc0 con 0x7fd5980a3cd0 2024-04-28T12:50:25.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.056+0000 7fd597fff700 1 -- 172.21.15.53:0/2484153331 <== mgr.14152 v2:172.21.15.53:6800/3118663054 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fd5a400e920 con 0x7fd5980a3cd0 2024-04-28T12:50:25.053 INFO:teuthology.orchestra.run.smithi053.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-28T12:50:25.053 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300G8480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-28T12:50:25.053 INFO:teuthology.orchestra.run.smithi053.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-28T12:50:25.053 INFO:teuthology.orchestra.run.smithi053.stdout:smithi191 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303008M400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-28T12:50:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd5980a3cd0 msgr2=0x7fd5980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd5980a3cd0 0x7fd5980a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4005fd0 tx=0x7fd5a4005e50 comp rx=0 tx=0).stop 2024-04-28T12:50:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 msgr2=0x7fd5b0104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:25.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b0104f70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800d010 tx=0x7fd5a8007ad0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 shutdown_connections 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fd5980a3cd0 0x7fd5980a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5b0103f20 0x7fd5b0104f70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b0102380 0x7fd5b01063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 --2- 172.21.15.53:0/2484153331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5b0103580 0x7fd5b0104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 >> 172.21.15.53:0/2484153331 conn(0x7fd5b00fd920 msgr2=0x7fd5b006b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 shutdown_connections 2024-04-28T12:50:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:25.059+0000 7fd5b501b700 1 -- 172.21.15.53:0/2484153331 wait complete. 2024-04-28T12:50:25.743 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-28T12:50:25.753 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-28T12:50:25.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-28T12:50:26.037 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:25 smithi191 ceph-mon[33016]: pgmap v343: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:25 smithi191 ceph-mon[33016]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:26.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:25 smithi053 ceph-mon[29182]: pgmap v343: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:26.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:25 smithi053 ceph-mon[29182]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:26.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:25 smithi077 ceph-mon[32553]: pgmap v343: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:26.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:25 smithi077 ceph-mon[32553]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:27.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.700+0000 7f88a72af700 1 -- 172.21.15.53:0/2601715078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0105020 msgr2=0x7f88a010b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.700+0000 7f88a72af700 1 --2- 172.21.15.53:0/2601715078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0105020 0x7f88a010b8d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f889800b0d0 tx=0x7f889800b3e0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.700+0000 7f88a72af700 1 -- 172.21.15.53:0/2601715078 shutdown_connections 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.700+0000 7f88a72af700 1 --2- 172.21.15.53:0/2601715078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0105020 0x7f88a010b8d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.700+0000 7f88a72af700 1 --2- 172.21.15.53:0/2601715078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88a0103480 0x7f88a01038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.700+0000 7f88a72af700 1 --2- 172.21.15.53:0/2601715078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a0104680 0x7f88a0104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.700+0000 7f88a72af700 1 -- 172.21.15.53:0/2601715078 >> 172.21.15.53:0/2601715078 conn(0x7f88a00fea20 msgr2=0x7f88a0100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.701+0000 7f88a72af700 1 -- 172.21.15.53:0/2601715078 shutdown_connections 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.701+0000 7f88a72af700 1 -- 172.21.15.53:0/2601715078 wait complete. 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.701+0000 7f88a72af700 1 Processor -- start 2024-04-28T12:50:27.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.701+0000 7f88a72af700 1 -- start start 2024-04-28T12:50:27.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a72af700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0103480 0x7f88a00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a72af700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a0104680 0x7f88a0075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a72af700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88a0105020 0x7f88a0075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a72af700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f88a010f010 con 0x7f88a0103480 2024-04-28T12:50:27.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a72af700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f88a010ee90 con 0x7f88a0105020 2024-04-28T12:50:27.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a72af700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f88a010f190 con 0x7f88a0104680 2024-04-28T12:50:27.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0103480 0x7f88a00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0103480 0x7f88a00771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41206/0 (socket says 172.21.15.53:41206) 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 -- 172.21.15.53:0/463454434 learned_addr learned my addr 172.21.15.53:0/463454434 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a484a700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a0104680 0x7f88a0075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a584c700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88a0105020 0x7f88a0075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 -- 172.21.15.53:0/463454434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a0104680 msgr2=0x7f88a0075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a0104680 0x7f88a0075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 -- 172.21.15.53:0/463454434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88a0105020 msgr2=0x7f88a0075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88a0105020 0x7f88a0075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 -- 172.21.15.53:0/463454434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a0076600 con 0x7f88a0103480 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a584c700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88a0105020 0x7f88a0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88a504b700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0103480 0x7f88a00771c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f889000b500 tx=0x7f889000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.702+0000 7f88967fc700 1 -- 172.21.15.53:0/463454434 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890015020 con 0x7f88a0103480 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.703+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a01a3850 con 0x7f88a0103480 2024-04-28T12:50:27.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.703+0000 7f88967fc700 1 -- 172.21.15.53:0/463454434 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8890011ce0 con 0x7f88a0103480 2024-04-28T12:50:27.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.703+0000 7f88967fc700 1 -- 172.21.15.53:0/463454434 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890010d60 con 0x7f88a0103480 2024-04-28T12:50:27.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.703+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a01a3d90 con 0x7f88a0103480 2024-04-28T12:50:27.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.704+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a006a110 con 0x7f88a0103480 2024-04-28T12:50:27.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.705+0000 7f88967fc700 1 -- 172.21.15.53:0/463454434 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8890018020 con 0x7f88a0103480 2024-04-28T12:50:27.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.705+0000 7f88967fc700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f888c0a3cd0 0x7f888c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.705+0000 7f88a484a700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f888c0a3cd0 0x7f888c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.706+0000 7f88967fc700 1 -- 172.21.15.53:0/463454434 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f88900cfd20 con 0x7f88a0103480 2024-04-28T12:50:27.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.706+0000 7f88a484a700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f888c0a3cd0 0x7f888c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f889c006fd0 tx=0x7f889c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:27.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.710+0000 7f88967fc700 1 -- 172.21.15.53:0/463454434 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f889001f020 con 0x7f88a0103480 2024-04-28T12:50:27.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.964+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f88a002e860 con 0x7f88a0103480 2024-04-28T12:50:27.961 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:27 smithi053 ceph-mon[29182]: pgmap v344: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.965+0000 7f88967fc700 1 -- 172.21.15.53:0/463454434 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) ==== 76+0+3050 (secure 0 0 0) 0x7f889009c6b0 con 0x7f88a0103480 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:e22 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:legacy client fscid: 1 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:Filesystem 'cephfs' (1) 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:fs_name cephfs 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 22 2024-04-28T12:50:27.962 INFO:teuthology.orchestra.run.smithi053.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:created 2024-04-28T12:48:43.873799+0000 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:modified 2024-04-28T12:48:49.455677+0000 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:tableserver 0 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:root 0 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:session_timeout 300 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:session_autoclose 300 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:max_file_size 1099511627776 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:max_xattr_size 65536 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:required_client_features {} 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:last_failure 0 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:last_failure_osd_epoch 0 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:50:27.963 INFO:teuthology.orchestra.run.smithi053.stdout:max_mds 3 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:in 0,1,2 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:up {0=24505,1=24485,2=24478} 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:failed 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:damaged 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:stopped 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:data_pools [3] 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:metadata_pool 2 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:inline_data disabled 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:balancer 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:bal_rank_mask -1 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:standby_count_wanted 0 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:qdb_cluster leader: 24505 members: 24505,24485,24478 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.i{0:24505} state up:active seq 5 addr [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.964 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.f{0:24469} state up:standby-replay seq 1 addr [v2:172.21.15.191:6834/2819920278,v1:172.21.15.191:6835/2819920278] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.h{1:24485} state up:active seq 7 addr [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.e{1:24455} state up:standby-replay seq 1 addr [v2:172.21.15.77:6834/1670611904,v1:172.21.15.77:6835/1670611904] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c{2:24478} state up:active seq 8 addr [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.k{2:24422} state up:standby-replay seq 1 addr [v2:172.21.15.77:6832/3944921810,v1:172.21.15.77:6833/3944921810] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:Standby daemons: 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.j{-1:14550} state up:standby seq 1 addr [v2:172.21.15.53:6834/1145263488,v1:172.21.15.53:6835/1145263488] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d{-1:14562} state up:standby seq 1 addr [v2:172.21.15.53:6836/3609735401,v1:172.21.15.53:6837/3609735401] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.g{-1:14616} state up:standby seq 1 addr [v2:172.21.15.53:6838/2476446586,v1:172.21.15.53:6839/2476446586] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a{-1:14628} state up:standby seq 1 addr [v2:172.21.15.53:6840/1978797303,v1:172.21.15.53:6841/1978797303] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b{-1:14646} state up:standby seq 1 addr [v2:172.21.15.77:6838/2957868311,v1:172.21.15.77:6839/2957868311] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.965 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.l{-1:24421} state up:standby seq 1 addr [v2:172.21.15.191:6832/1697828385,v1:172.21.15.191:6833/1697828385] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:50:27.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f888c0a3cd0 msgr2=0x7f888c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f888c0a3cd0 0x7f888c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f889c006fd0 tx=0x7f889c008040 comp rx=0 tx=0).stop 2024-04-28T12:50:27.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0103480 msgr2=0x7f88a00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0103480 0x7f88a00771c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f889000b500 tx=0x7f889000b8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 shutdown_connections 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f888c0a3cd0 0x7f888c0a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88a0103480 0x7f88a00771c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88a0105020 0x7f88a0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 --2- 172.21.15.53:0/463454434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88a0104680 0x7f88a0075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 >> 172.21.15.53:0/463454434 conn(0x7f88a00fea20 msgr2=0x7f88a0107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 shutdown_connections 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:27.967+0000 7f88a72af700 1 -- 172.21.15.53:0/463454434 wait complete. 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 22 2024-04-28T12:50:28.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:27 smithi191 ceph-mon[33016]: pgmap v344: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:50:28.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:27 smithi077 ceph-mon[32553]: pgmap v344: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T12:50:28.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10525726-055c-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-28T12:50:28.975 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:29.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:28 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/463454434' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T12:50:29.097 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:28 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/463454434' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T12:50:29.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:28 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/463454434' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T12:50:30.036 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:29 smithi053 ceph-mon[29182]: pgmap v345: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:30.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:29 smithi191 ceph-mon[33016]: pgmap v345: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:30.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:29 smithi077 ceph-mon[32553]: pgmap v345: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 -- 172.21.15.53:0/1881123601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 msgr2=0x7ff570104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 --2- 172.21.15.53:0/1881123601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570104b00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff5640099c0 tx=0x7ff564009cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 -- 172.21.15.53:0/1881123601 shutdown_connections 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 --2- 172.21.15.53:0/1881123601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff570105040 0x7ff57010b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 --2- 172.21.15.53:0/1881123601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5701034a0 0x7ff5701038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 --2- 172.21.15.53:0/1881123601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570104b00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 -- 172.21.15.53:0/1881123601 >> 172.21.15.53:0/1881123601 conn(0x7ff5700fea40 msgr2=0x7ff570100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 -- 172.21.15.53:0/1881123601 shutdown_connections 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.584+0000 7ff5749a4700 1 -- 172.21.15.53:0/1881123601 wait complete. 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 Processor -- start 2024-04-28T12:50:30.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 -- start start 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5701034a0 0x7ff5700771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff570105040 0x7ff570075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff57010ef20 con 0x7ff5701046a0 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff57010eda0 con 0x7ff5701034a0 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff5749a4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff57010f0a0 con 0x7ff570105040 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570075810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41232/0 (socket says 172.21.15.53:41232) 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 -- 172.21.15.53:0/2386016444 learned_addr learned my addr 172.21.15.53:0/2386016444 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56e59c700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5701034a0 0x7ff5700771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56ed9d700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff570105040 0x7ff570075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 -- 172.21.15.53:0/2386016444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff570105040 msgr2=0x7ff570075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff570105040 0x7ff570075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 -- 172.21.15.53:0/2386016444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5701034a0 msgr2=0x7ff5700771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5701034a0 0x7ff5700771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.585+0000 7ff56dd9b700 1 -- 172.21.15.53:0/2386016444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff570076600 con 0x7ff5701046a0 2024-04-28T12:50:30.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.586+0000 7ff56dd9b700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570075810 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff56400b580 tx=0x7ff564009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:30.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.586+0000 7ff55f7fe700 1 -- 172.21.15.53:0/2386016444 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564027070 con 0x7ff5701046a0 2024-04-28T12:50:30.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.586+0000 7ff55f7fe700 1 -- 172.21.15.53:0/2386016444 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5640040a0 con 0x7ff5701046a0 2024-04-28T12:50:30.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.586+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5701a3660 con 0x7ff5701046a0 2024-04-28T12:50:30.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.586+0000 7ff55f7fe700 1 -- 172.21.15.53:0/2386016444 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564022ce0 con 0x7ff5701046a0 2024-04-28T12:50:30.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.586+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5701a3b10 con 0x7ff5701046a0 2024-04-28T12:50:30.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.587+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57006a110 con 0x7ff5701046a0 2024-04-28T12:50:30.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.588+0000 7ff55f7fe700 1 -- 172.21.15.53:0/2386016444 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff56402a070 con 0x7ff5701046a0 2024-04-28T12:50:30.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.589+0000 7ff55f7fe700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff5580a3cd0 0x7ff5580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.589+0000 7ff56e59c700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff5580a3cd0 0x7ff5580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.589+0000 7ff55f7fe700 1 -- 172.21.15.53:0/2386016444 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7ff5640d11c0 con 0x7ff5701046a0 2024-04-28T12:50:30.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.589+0000 7ff56e59c700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff5580a3cd0 0x7ff5580a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff570104500 tx=0x7ff56000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:30.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.593+0000 7ff55f7fe700 1 -- 172.21.15.53:0/2386016444 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff564013070 con 0x7ff5701046a0 2024-04-28T12:50:30.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.801+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7ff57002e490 con 0x7ff5701046a0 2024-04-28T12:50:30.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.802+0000 7ff55f7fe700 1 -- 172.21.15.53:0/2386016444 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5298 (secure 0 0 0) 0x7ff56409dbf0 con 0x7ff5701046a0 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 88 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:fsid 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:created 2024-04-28T12:40:39.137641+0000 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:modified 2024-04-28T12:49:35.625293+0000 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:crush_version 26 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:full_ratio 0.95 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:backfillfull_ratio 0.9 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:nearfull_ratio 0.85 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:require_min_compat_client luminous 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:min_compat_client luminous 2024-04-28T12:50:30.799 INFO:teuthology.orchestra.run.smithi053.stdout:require_osd_release squid 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:stretch_mode_enabled false 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:max_osd 12 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6802/1076361705,v1:172.21.15.53:6803/1076361705] [v2:172.21.15.53:6804/1076361705,v1:172.21.15.53:6805/1076361705] exists,up 2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6800/2126226610,v1:172.21.15.77:6801/2126226610] [v2:172.21.15.77:6802/2126226610,v1:172.21.15.77:6803/2126226610] exists,up f15997c2-61f7-4970-b488-c38990780d97 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/246351477,v1:172.21.15.191:6801/246351477] [v2:172.21.15.191:6802/246351477,v1:172.21.15.191:6803/246351477] exists,up 129b78c4-4920-41b7-bf2e-b512e63ae231 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 up in weight 1 up_from 26 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6810/3076698859,v1:172.21.15.53:6811/3076698859] [v2:172.21.15.53:6812/3076698859,v1:172.21.15.53:6813/3076698859] exists,up cbade960-1344-45b7-a7ed-a056a7500a6f 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6808/3997878075,v1:172.21.15.77:6809/3997878075] [v2:172.21.15.77:6810/3997878075,v1:172.21.15.77:6811/3997878075] exists,up 990ad0bc-63b6-4466-94ca-75ba51801c7c 2024-04-28T12:50:30.800 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 up in weight 1 up_from 39 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/2749306755,v1:172.21.15.191:6809/2749306755] [v2:172.21.15.191:6810/2749306755,v1:172.21.15.191:6811/2749306755] exists,up 3015dc0f-1a0e-4fc9-b38c-f779c500f3b5 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6818/512872966,v1:172.21.15.53:6819/512872966] [v2:172.21.15.53:6820/512872966,v1:172.21.15.53:6821/512872966] exists,up 2591cbe0-d151-45b4-87c1-532c13a53037 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6816/1250050488,v1:172.21.15.77:6817/1250050488] [v2:172.21.15.77:6818/1250050488,v1:172.21.15.77:6819/1250050488] exists,up 573bb4c3-4e03-48c7-ad54-90cc71ab0311 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.8 up in weight 1 up_from 60 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/1864030319,v1:172.21.15.191:6817/1864030319] [v2:172.21.15.191:6818/1864030319,v1:172.21.15.191:6819/1864030319] exists,up 44f526cb-dce8-4894-9cd7-21118f8d3052 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.9 up in weight 1 up_from 68 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6826/3727947776,v1:172.21.15.53:6827/3727947776] [v2:172.21.15.53:6828/3727947776,v1:172.21.15.53:6829/3727947776] exists,up aec50a3e-2db8-4933-812d-3ac318ddccdf 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6824/2556585572,v1:172.21.15.77:6825/2556585572] [v2:172.21.15.77:6826/2556585572,v1:172.21.15.77:6827/2556585572] exists,up e1817077-d7a5-4eeb-8e2a-b948147d22a3 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/1302539827,v1:172.21.15.191:6825/1302539827] [v2:172.21.15.191:6826/1302539827,v1:172.21.15.191:6827/1302539827] exists,up 4c0e510f-d109-48e1-844c-1e2146a8a412 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 2.5 [8,11] 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 2.b [2,1] 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 2.14 [2,4] 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 2.1d [8,11] 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.6 [4,7] 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.20 [5,9] 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.22 [3,9] 2024-04-28T12:50:30.801 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.31 [4,8] 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.36 [4,9] 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6800/2275002354 expires 2024-04-29T12:41:34.367887+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/3169008100 expires 2024-04-29T12:41:34.367887+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6801/3936438773 expires 2024-04-29T12:41:06.674945+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/3416041324 expires 2024-04-29T12:41:34.367887+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/1704277076 expires 2024-04-29T12:41:34.367887+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/591838983 expires 2024-04-29T12:41:06.674945+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6801/2275002354 expires 2024-04-29T12:41:34.367887+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/2809885884 expires 2024-04-29T12:41:06.674945+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/1598561968 expires 2024-04-29T12:41:06.674945+0000 2024-04-28T12:50:30.802 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6800/3936438773 expires 2024-04-29T12:41:06.674945+0000 2024-04-28T12:50:30.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff5580a3cd0 msgr2=0x7ff5580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff5580a3cd0 0x7ff5580a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff570104500 tx=0x7ff56000a3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 msgr2=0x7ff570075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570075810 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff56400b580 tx=0x7ff564009510 comp rx=0 tx=0).stop 2024-04-28T12:50:30.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 shutdown_connections 2024-04-28T12:50:30.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff5580a3cd0 0x7ff5580a6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff570105040 0x7ff570075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5701034a0 0x7ff5700771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 --2- 172.21.15.53:0/2386016444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5701046a0 0x7ff570075810 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.804+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 >> 172.21.15.53:0/2386016444 conn(0x7ff5700fea40 msgr2=0x7ff5701079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:30.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.805+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 shutdown_connections 2024-04-28T12:50:30.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:30.805+0000 7ff5749a4700 1 -- 172.21.15.53:0/2386016444 wait complete. 2024-04-28T12:50:31.605 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-28T12:50:31.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:31.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 -- 172.21.15.53:0/566740281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94103420 msgr2=0x7fbf94109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/566740281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94103420 0x7fbf94109cb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009900 tx=0x7fbf8c02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 -- 172.21.15.53:0/566740281 shutdown_connections 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/566740281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94103420 0x7fbf94109cb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/566740281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94102a60 0x7fbf94102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/566740281 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf94101870 0x7fbf94101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 -- 172.21.15.53:0/566740281 >> 172.21.15.53:0/566740281 conn(0x7fbf940fcde0 msgr2=0x7fbf940ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 -- 172.21.15.53:0/566740281 shutdown_connections 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 -- 172.21.15.53:0/566740281 wait complete. 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.731+0000 7fbf9aec8640 1 Processor -- start 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf9aec8640 1 -- start start 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf9aec8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf94101870 0x7fbf9406a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf9aec8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94102a60 0x7fbf9419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf9aec8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94103420 0x7fbf9419fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf9aec8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410d380 con 0x7fbf94102a60 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf9aec8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410d200 con 0x7fbf94103420 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf9aec8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410d500 con 0x7fbf94101870 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94102a60 0x7fbf9419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94102a60 0x7fbf9419d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41246/0 (socket says 172.21.15.53:41246) 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 -- 172.21.15.53:0/3920147150 learned_addr learned my addr 172.21.15.53:0/3920147150 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf98c64640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf94101870 0x7fbf9406a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf99465640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94103420 0x7fbf9419fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 -- 172.21.15.53:0/3920147150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf94101870 msgr2=0x7fbf9406a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf94101870 0x7fbf9406a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 -- 172.21.15.53:0/3920147150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94103420 msgr2=0x7fbf9419fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94103420 0x7fbf9419fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 -- 172.21.15.53:0/3920147150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf941a0750 con 0x7fbf94102a60 2024-04-28T12:50:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf99465640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94103420 0x7fbf9419fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:31.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf93fff640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94102a60 0x7fbf9419d820 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400eab0 tx=0x7fbf8400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:31.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.732+0000 7fbf91ffb640 1 -- 172.21.15.53:0/3920147150 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8400ce40 con 0x7fbf94102a60 2024-04-28T12:50:31.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.733+0000 7fbf91ffb640 1 -- 172.21.15.53:0/3920147150 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf84011920 con 0x7fbf94102a60 2024-04-28T12:50:31.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.733+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf941a09e0 con 0x7fbf94102a60 2024-04-28T12:50:31.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.733+0000 7fbf91ffb640 1 -- 172.21.15.53:0/3920147150 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84010a30 con 0x7fbf94102a60 2024-04-28T12:50:31.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.733+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf941a0f30 con 0x7fbf94102a60 2024-04-28T12:50:31.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.734+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf94108b90 con 0x7fbf94102a60 2024-04-28T12:50:31.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.735+0000 7fbf91ffb640 1 -- 172.21.15.53:0/3920147150 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbf84010bd0 con 0x7fbf94102a60 2024-04-28T12:50:31.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.735+0000 7fbf91ffb640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbf800a3c40 0x7fbf800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.736+0000 7fbf98c64640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbf800a3c40 0x7fbf800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.736+0000 7fbf91ffb640 1 -- 172.21.15.53:0/3920147150 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fbf840d0c90 con 0x7fbf94102a60 2024-04-28T12:50:31.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.736+0000 7fbf98c64640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbf800a3c40 0x7fbf800a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0058c0 tx=0x7fbf7c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:31.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.740+0000 7fbf91ffb640 1 -- 172.21.15.53:0/3920147150 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbf840d6020 con 0x7fbf94102a60 2024-04-28T12:50:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:31 smithi053 ceph-mon[29182]: pgmap v346: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:31.895 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:31 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2386016444' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T12:50:31.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.928+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf9402dad0 con 0x7fbf94102a60 2024-04-28T12:50:31.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.930+0000 7fbf91ffb640 1 -- 172.21.15.53:0/3920147150 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7fbf84016030 con 0x7fbf94102a60 2024-04-28T12:50:31.927 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:31.927 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbf800a3c40 msgr2=0x7fbf800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbf800a3c40 0x7fbf800a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0058c0 tx=0x7fbf7c005850 comp rx=0 tx=0).stop 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94102a60 msgr2=0x7fbf9419d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94102a60 0x7fbf9419d820 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400eab0 tx=0x7fbf8400ef70 comp rx=0 tx=0).stop 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 shutdown_connections 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbf800a3c40 0x7fbf800a6100 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94103420 0x7fbf9419fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf94102a60 0x7fbf9419d820 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 --2- 172.21.15.53:0/3920147150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf94101870 0x7fbf9406a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 >> 172.21.15.53:0/3920147150 conn(0x7fbf940fcde0 msgr2=0x7fbf94105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 shutdown_connections 2024-04-28T12:50:31.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:31.932+0000 7fbf9aec8640 1 -- 172.21.15.53:0/3920147150 wait complete. 2024-04-28T12:50:31.940 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:42.284772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:42.832857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:50:31.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:50:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:31 smithi191 ceph-mon[33016]: pgmap v346: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:32.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:31 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2386016444' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T12:50:32.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.049+0000 7f7de9893640 1 -- 172.21.15.53:0/4179836921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4101870 msgr2=0x7f7de4101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.049+0000 7f7de9893640 1 --2- 172.21.15.53:0/4179836921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4101870 0x7f7de4101c90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009900 tx=0x7f7ddc02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:32.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.049+0000 7f7de9893640 1 -- 172.21.15.53:0/4179836921 shutdown_connections 2024-04-28T12:50:32.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.049+0000 7f7de9893640 1 --2- 172.21.15.53:0/4179836921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de4103420 0x7f7de4109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 --2- 172.21.15.53:0/4179836921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de4102a60 0x7f7de4102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 --2- 172.21.15.53:0/4179836921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4101870 0x7f7de4101c90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 -- 172.21.15.53:0/4179836921 >> 172.21.15.53:0/4179836921 conn(0x7f7de40fcde0 msgr2=0x7f7de40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 -- 172.21.15.53:0/4179836921 shutdown_connections 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 -- 172.21.15.53:0/4179836921 wait complete. 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 Processor -- start 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 -- start start 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.050+0000 7f7de9893640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de4101870 0x7f7de4105800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de9893640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de4102a60 0x7f7de4103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de9893640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4103420 0x7f7de4104380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de9893640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410d390 con 0x7f7de4103420 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de9893640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410d210 con 0x7f7de4101870 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de9893640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410d510 con 0x7f7de4102a60 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4103420 0x7f7de4104380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4103420 0x7f7de4104380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41266/0 (socket says 172.21.15.53:41266) 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 -- 172.21.15.53:0/2321045031 learned_addr learned my addr 172.21.15.53:0/2321045031 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de2ffd640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de4101870 0x7f7de4105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de27fc640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de4102a60 0x7f7de4103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 -- 172.21.15.53:0/2321045031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de4102a60 msgr2=0x7f7de4103e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de4102a60 0x7f7de4103e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 -- 172.21.15.53:0/2321045031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de4101870 msgr2=0x7f7de4105800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de4101870 0x7f7de4105800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 -- 172.21.15.53:0/2321045031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de4104c10 con 0x7f7de4103420 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de2ffd640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de4101870 0x7f7de4105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de37fe640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4103420 0x7f7de4104380 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009fd0 tx=0x7f7dd400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de8891640 1 -- 172.21.15.53:0/2321045031 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4017070 con 0x7f7de4103420 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de41a2d60 con 0x7f7de4103420 2024-04-28T12:50:32.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de8891640 1 -- 172.21.15.53:0/2321045031 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dd4003e20 con 0x7f7de4103420 2024-04-28T12:50:32.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de8891640 1 -- 172.21.15.53:0/2321045031 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40106c0 con 0x7f7de4103420 2024-04-28T12:50:32.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.051+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de41a31a0 con 0x7f7de4103420 2024-04-28T12:50:32.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.052+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de41a3430 con 0x7f7de4103420 2024-04-28T12:50:32.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.053+0000 7f7de8891640 1 -- 172.21.15.53:0/2321045031 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7dd4004140 con 0x7f7de4103420 2024-04-28T12:50:32.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.054+0000 7f7de8891640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7dcc0a3c40 0x7f7dcc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.054+0000 7f7de2ffd640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7dcc0a3c40 0x7f7dcc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.054+0000 7f7de8891640 1 -- 172.21.15.53:0/2321045031 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f7dd40d0280 con 0x7f7de4103420 2024-04-28T12:50:32.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.055+0000 7f7de2ffd640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7dcc0a3c40 0x7f7dcc0a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009230 tx=0x7f7ddc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.059+0000 7f7de8891640 1 -- 172.21.15.53:0/2321045031 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7dd40047e0 con 0x7f7de4103420 2024-04-28T12:50:32.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:31 smithi077 ceph-mon[32553]: pgmap v346: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:32.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:31 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2386016444' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T12:50:32.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.243+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7de41a37c0 con 0x7f7de4103420 2024-04-28T12:50:32.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.243+0000 7f7de8891640 1 -- 172.21.15.53:0/2321045031 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f7dd409cb00 con 0x7f7de4103420 2024-04-28T12:50:32.240 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7dcc0a3c40 msgr2=0x7f7dcc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7dcc0a3c40 0x7f7dcc0a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009230 tx=0x7f7ddc0314a0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4103420 msgr2=0x7f7de4104380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4103420 0x7f7de4104380 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009fd0 tx=0x7f7dd400b260 comp rx=0 tx=0).stop 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 shutdown_connections 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f7dcc0a3c40 0x7f7dcc0a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4103420 0x7f7de4104380 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7de4102a60 0x7f7de4103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 --2- 172.21.15.53:0/2321045031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de4101870 0x7f7de4105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 >> 172.21.15.53:0/2321045031 conn(0x7f7de40fcde0 msgr2=0x7f7de4069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 shutdown_connections 2024-04-28T12:50:32.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.245+0000 7f7de9893640 1 -- 172.21.15.53:0/2321045031 wait complete. 2024-04-28T12:50:32.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:50:32.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 -- 172.21.15.53:0/1916981258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102e10 msgr2=0x7f0c00103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 --2- 172.21.15.53:0/1916981258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102e10 0x7f0c00103230 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009900 tx=0x7f0bf802a410 comp rx=0 tx=0).stop 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 -- 172.21.15.53:0/1916981258 shutdown_connections 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 --2- 172.21.15.53:0/1916981258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c00102500 0x7f0c00109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 --2- 172.21.15.53:0/1916981258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c00101b40 0x7f0c00101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 --2- 172.21.15.53:0/1916981258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102e10 0x7f0c00103230 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 -- 172.21.15.53:0/1916981258 >> 172.21.15.53:0/1916981258 conn(0x7f0c000fc570 msgr2=0x7f0c000fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 -- 172.21.15.53:0/1916981258 shutdown_connections 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.361+0000 7f0c08789640 1 -- 172.21.15.53:0/1916981258 wait complete. 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 Processor -- start 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 -- start start 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c00101b40 0x7f0c00198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102500 0x7f0c001991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c00102e10 0x7f0c001a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0010d3e0 con 0x7f0c00102500 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0010d260 con 0x7f0c00101b40 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c08789640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0010d560 con 0x7f0c00102e10 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c05d24640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102500 0x7f0c001991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c05d24640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102500 0x7f0c001991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41294/0 (socket says 172.21.15.53:41294) 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.362+0000 7f0c05d24640 1 -- 172.21.15.53:0/3825251589 learned_addr learned my addr 172.21.15.53:0/3825251589 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c06525640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c00101b40 0x7f0c00198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c05d24640 1 -- 172.21.15.53:0/3825251589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c00102e10 msgr2=0x7f0c001a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c05d24640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c00102e10 0x7f0c001a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c05d24640 1 -- 172.21.15.53:0/3825251589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c00101b40 msgr2=0x7f0c00198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c05d24640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c00101b40 0x7f0c00198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c05d24640 1 -- 172.21.15.53:0/3825251589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c001a0890 con 0x7f0c00102500 2024-04-28T12:50:32.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c06525640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c00101b40 0x7f0c00198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c05d24640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102500 0x7f0c001991c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400d950 tx=0x7f0bf400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0bff7fe640 1 -- 172.21.15.53:0/3825251589 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4017070 con 0x7f0c00102500 2024-04-28T12:50:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0bff7fe640 1 -- 172.21.15.53:0/3825251589 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bf4009c30 con 0x7f0c00102500 2024-04-28T12:50:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c001a0b80 con 0x7f0c00102500 2024-04-28T12:50:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0bff7fe640 1 -- 172.21.15.53:0/3825251589 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf400f620 con 0x7f0c00102500 2024-04-28T12:50:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.363+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c001a10d0 con 0x7f0c00102500 2024-04-28T12:50:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.364+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c001a12a0 con 0x7f0c00102500 2024-04-28T12:50:32.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.365+0000 7f0bff7fe640 1 -- 172.21.15.53:0/3825251589 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0bf4010460 con 0x7f0c00102500 2024-04-28T12:50:32.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.366+0000 7f0bff7fe640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0be80a3c40 0x7f0be80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.366+0000 7f0bff7fe640 1 -- 172.21.15.53:0/3825251589 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f0bf40d02e0 con 0x7f0c00102500 2024-04-28T12:50:32.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.366+0000 7f0c06525640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0be80a3c40 0x7f0be80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.366+0000 7f0c06525640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0be80a3c40 0x7f0be80a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0bf802a930 tx=0x7f0bf8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.369+0000 7f0bff7fe640 1 -- 172.21.15.53:0/3825251589 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0bf409cb60 con 0x7f0c00102500 2024-04-28T12:50:32.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.555+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0c001a1550 con 0x7f0c00102500 2024-04-28T12:50:32.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.556+0000 7f0bff7fe640 1 -- 172.21.15.53:0/3825251589 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f0bf4009de0 con 0x7f0c00102500 2024-04-28T12:50:32.552 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0be80a3c40 msgr2=0x7f0be80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0be80a3c40 0x7f0be80a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0bf802a930 tx=0x7f0bf8000bc0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102500 msgr2=0x7f0c001991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102500 0x7f0c001991c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400d950 tx=0x7f0bf400de10 comp rx=0 tx=0).stop 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 shutdown_connections 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0be80a3c40 0x7f0be80a6100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c00102e10 0x7f0c001a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c00102500 0x7f0c001991c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 --2- 172.21.15.53:0/3825251589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c00101b40 0x7f0c00198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 >> 172.21.15.53:0/3825251589 conn(0x7f0c000fc570 msgr2=0x7f0c00100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 shutdown_connections 2024-04-28T12:50:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.558+0000 7f0c08789640 1 -- 172.21.15.53:0/3825251589 wait complete. 2024-04-28T12:50:32.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:50:32.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 -- 172.21.15.53:0/467598432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4104540 msgr2=0x7f90e410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 --2- 172.21.15.53:0/467598432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4104540 0x7f90e410add0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009ca0 tx=0x7f90dc02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 -- 172.21.15.53:0/467598432 shutdown_connections 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 --2- 172.21.15.53:0/467598432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4104540 0x7f90e410add0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 --2- 172.21.15.53:0/467598432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4103b80 0x7f90e4104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 --2- 172.21.15.53:0/467598432 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90e4102990 0x7f90e4102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 -- 172.21.15.53:0/467598432 >> 172.21.15.53:0/467598432 conn(0x7f90e40fdf00 msgr2=0x7f90e4100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 -- 172.21.15.53:0/467598432 shutdown_connections 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 -- 172.21.15.53:0/467598432 wait complete. 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 Processor -- start 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.674+0000 7f90eb591640 1 -- start start 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4102990 0x7f90e4198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4103b80 0x7f90e41990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90e4104540 0x7f90e41a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410e4c0 con 0x7f90e4102990 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410e340 con 0x7f90e4103b80 2024-04-28T12:50:32.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410e640 con 0x7f90e4104540 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4102990 0x7f90e4198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4102990 0x7f90e4198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41300/0 (socket says 172.21.15.53:41300) 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 -- 172.21.15.53:0/2820073636 learned_addr learned my addr 172.21.15.53:0/2820073636 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e8b2c640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4103b80 0x7f90e41990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e9b2e640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90e4104540 0x7f90e41a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 -- 172.21.15.53:0/2820073636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90e4104540 msgr2=0x7f90e41a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90e4104540 0x7f90e41a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 -- 172.21.15.53:0/2820073636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4103b80 msgr2=0x7f90e41990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4103b80 0x7f90e41990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 -- 172.21.15.53:0/2820073636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e41a0790 con 0x7f90e4102990 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e9b2e640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90e4104540 0x7f90e41a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90e932d640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4102990 0x7f90e4198b70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f90e000d950 tx=0x7f90e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90da7fc640 1 -- 172.21.15.53:0/2820073636 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e0017070 con 0x7f90e4102990 2024-04-28T12:50:32.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e41a0a80 con 0x7f90e4102990 2024-04-28T12:50:32.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90da7fc640 1 -- 172.21.15.53:0/2820073636 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90e0009c30 con 0x7f90e4102990 2024-04-28T12:50:32.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90da7fc640 1 -- 172.21.15.53:0/2820073636 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e000f620 con 0x7f90e4102990 2024-04-28T12:50:32.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.675+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e41a0f80 con 0x7f90e4102990 2024-04-28T12:50:32.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.676+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e4109cb0 con 0x7f90e4102990 2024-04-28T12:50:32.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.678+0000 7f90da7fc640 1 -- 172.21.15.53:0/2820073636 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f90e001a070 con 0x7f90e4102990 2024-04-28T12:50:32.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.679+0000 7f90da7fc640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f90d00a3c40 0x7f90d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.679+0000 7f90e8b2c640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f90d00a3c40 0x7f90d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.679+0000 7f90da7fc640 1 -- 172.21.15.53:0/2820073636 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f90e00cfff0 con 0x7f90e4102990 2024-04-28T12:50:32.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.679+0000 7f90e8b2c640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f90d00a3c40 0x7f90d00a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f90e419a080 tx=0x7f90d400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.682+0000 7f90da7fc640 1 -- 172.21.15.53:0/2820073636 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f90e0009de0 con 0x7f90e4102990 2024-04-28T12:50:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.869+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f90e402dc60 con 0x7f90e4102990 2024-04-28T12:50:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.869+0000 7f90da7fc640 1 -- 172.21.15.53:0/2820073636 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f90e009c830 con 0x7f90e4102990 2024-04-28T12:50:32.866 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f90d00a3c40 msgr2=0x7f90d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f90d00a3c40 0x7f90d00a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f90e419a080 tx=0x7f90d400b390 comp rx=0 tx=0).stop 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4102990 msgr2=0x7f90e4198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4102990 0x7f90e4198b70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f90e000d950 tx=0x7f90e000de10 comp rx=0 tx=0).stop 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 shutdown_connections 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f90d00a3c40 0x7f90d00a6100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90e4104540 0x7f90e41a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4103b80 0x7f90e41990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 --2- 172.21.15.53:0/2820073636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90e4102990 0x7f90e4198b70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 >> 172.21.15.53:0/2820073636 conn(0x7f90e40fdf00 msgr2=0x7f90e4106e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 shutdown_connections 2024-04-28T12:50:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.871+0000 7f90eb591640 1 -- 172.21.15.53:0/2820073636 wait complete. 2024-04-28T12:50:32.878 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:32 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3920147150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:32.878 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:32 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2321045031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:32.878 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:32 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3825251589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:32.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:32.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 -- 172.21.15.53:0/1634220528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 msgr2=0x7faa3c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 --2- 172.21.15.53:0/1634220528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c101cb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faa38009900 tx=0x7faa3802a410 comp rx=0 tx=0).stop 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 -- 172.21.15.53:0/1634220528 shutdown_connections 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 --2- 172.21.15.53:0/1634220528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3c103440 0x7faa3c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 --2- 172.21.15.53:0/1634220528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa3c102a80 0x7faa3c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 --2- 172.21.15.53:0/1634220528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c101cb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 -- 172.21.15.53:0/1634220528 >> 172.21.15.53:0/1634220528 conn(0x7faa3c0fce20 msgr2=0x7faa3c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 -- 172.21.15.53:0/1634220528 shutdown_connections 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.984+0000 7faa43d19640 1 -- 172.21.15.53:0/1634220528 wait complete. 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 Processor -- start 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 -- start start 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3c102a80 0x7faa3c199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa3c103440 0x7faa3c1a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10d410 con 0x7faa3c101890 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10d290 con 0x7faa3c103440 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa43d19640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10d590 con 0x7faa3c102a80 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa41ab5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa41ab5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41310/0 (socket says 172.21.15.53:41310) 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa41ab5640 1 -- 172.21.15.53:0/1662814063 learned_addr learned my addr 172.21.15.53:0/1662814063 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa412b4640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3c102a80 0x7faa3c199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa422b6640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa3c103440 0x7faa3c1a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa41ab5640 1 -- 172.21.15.53:0/1662814063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3c102a80 msgr2=0x7faa3c199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa41ab5640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3c102a80 0x7faa3c199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa41ab5640 1 -- 172.21.15.53:0/1662814063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa3c103440 msgr2=0x7faa3c1a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.985+0000 7faa41ab5640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa3c103440 0x7faa3c1a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa41ab5640 1 -- 172.21.15.53:0/1662814063 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c1a07c0 con 0x7faa3c101890 2024-04-28T12:50:32.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa422b6640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa3c103440 0x7faa3c1a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:32.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa41ab5640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c198c10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa38006010 tx=0x7faa38004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa32ffd640 1 -- 172.21.15.53:0/1662814063 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3803f030 con 0x7faa3c101890 2024-04-28T12:50:32.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa32ffd640 1 -- 172.21.15.53:0/1662814063 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa38003cb0 con 0x7faa3c101890 2024-04-28T12:50:32.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa32ffd640 1 -- 172.21.15.53:0/1662814063 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa38035070 con 0x7faa3c101890 2024-04-28T12:50:32.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3c1a0a50 con 0x7faa3c101890 2024-04-28T12:50:32.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.986+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa3c1a0f90 con 0x7faa3c101890 2024-04-28T12:50:32.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.987+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa3c1a1220 con 0x7faa3c101890 2024-04-28T12:50:32.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.989+0000 7faa32ffd640 1 -- 172.21.15.53:0/1662814063 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faa38042030 con 0x7faa3c101890 2024-04-28T12:50:32.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.989+0000 7faa32ffd640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa280a3cc0 0x7faa280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:32.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.989+0000 7faa412b4640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa280a3cc0 0x7faa280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:32.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.989+0000 7faa32ffd640 1 -- 172.21.15.53:0/1662814063 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7faa380f10b0 con 0x7faa3c101890 2024-04-28T12:50:32.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.990+0000 7faa412b4640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa280a3cc0 0x7faa280a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faa2c005e10 tx=0x7faa2c005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:32.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:32.993+0000 7faa32ffd640 1 -- 172.21.15.53:0/1662814063 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faa380bd930 con 0x7faa3c101890 2024-04-28T12:50:33.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:32 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3920147150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:32 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2321045031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:33.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:32 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3825251589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:32 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3920147150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:33.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:32 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2321045031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:33.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:32 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3825251589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:33.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.217+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa3c1a13f0 con 0x7faa3c101890 2024-04-28T12:50:33.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.218+0000 7faa32ffd640 1 -- 172.21.15.53:0/1662814063 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11873 (secure 0 0 0) 0x7faa38003e60 con 0x7faa3c101890 2024-04-28T12:50:33.215 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:33.215 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:48:49.455677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24485,24478]},"id":1}]} 2024-04-28T12:50:33.215 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 22 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa280a3cc0 msgr2=0x7faa280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa280a3cc0 0x7faa280a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faa2c005e10 tx=0x7faa2c005da0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 msgr2=0x7faa3c198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c198c10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa38006010 tx=0x7faa38004430 comp rx=0 tx=0).stop 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 shutdown_connections 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7faa280a3cc0 0x7faa280a6180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa3c103440 0x7faa3c1a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3c102a80 0x7faa3c199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 --2- 172.21.15.53:0/1662814063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa3c101890 0x7faa3c198c10 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 >> 172.21.15.53:0/1662814063 conn(0x7faa3c0fce20 msgr2=0x7faa3c0fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 shutdown_connections 2024-04-28T12:50:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.220+0000 7faa43d19640 1 -- 172.21.15.53:0/1662814063 wait complete. 2024-04-28T12:50:33.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 -- 172.21.15.53:0/1731176053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48103410 msgr2=0x7f8a48105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/1731176053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48103410 0x7f8a48105800 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009900 tx=0x7f8a4402a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 -- 172.21.15.53:0/1731176053 shutdown_connections 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/1731176053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a48108670 0x7f8a4810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/1731176053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a48105d40 0x7f8a48108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/1731176053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48103410 0x7f8a48105800 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 -- 172.21.15.53:0/1731176053 >> 172.21.15.53:0/1731176053 conn(0x7f8a480fce20 msgr2=0x7f8a480ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 -- 172.21.15.53:0/1731176053 shutdown_connections 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.333+0000 7f8a4f70a640 1 -- 172.21.15.53:0/1731176053 wait complete. 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 Processor -- start 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 -- start start 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a48103410 0x7f8a48102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48105d40 0x7f8a481013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a48108670 0x7f8a48101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4810e4c0 con 0x7f8a48105d40 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4810e340 con 0x7f8a48103410 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4f70a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4810e640 con 0x7f8a48108670 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48105d40 0x7f8a481013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48105d40 0x7f8a481013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41320/0 (socket says 172.21.15.53:41320) 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 -- 172.21.15.53:0/78544113 learned_addr learned my addr 172.21.15.53:0/78544113 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4d4a6640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a48103410 0x7f8a48102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4dca7640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a48108670 0x7f8a48101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 -- 172.21.15.53:0/78544113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a48108670 msgr2=0x7f8a48101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a48108670 0x7f8a48101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 -- 172.21.15.53:0/78544113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a48103410 msgr2=0x7f8a48102d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a48103410 0x7f8a48102d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 -- 172.21.15.53:0/78544113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a48102190 con 0x7f8a48105d40 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4dca7640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a48108670 0x7f8a48101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4d4a6640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a48103410 0x7f8a48102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a4cca5640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48105d40 0x7f8a481013c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800c970 tx=0x7f8a3800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.334+0000 7f8a3e7fc640 1 -- 172.21.15.53:0/78544113 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38014070 con 0x7f8a48105d40 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.335+0000 7f8a3e7fc640 1 -- 172.21.15.53:0/78544113 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a38004440 con 0x7f8a48105d40 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.335+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a481a2c30 con 0x7f8a48105d40 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.335+0000 7f8a3e7fc640 1 -- 172.21.15.53:0/78544113 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38003e50 con 0x7f8a48105d40 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.335+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a481a3180 con 0x7f8a48105d40 2024-04-28T12:50:33.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.336+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a480fe910 con 0x7f8a48105d40 2024-04-28T12:50:33.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.337+0000 7f8a3e7fc640 1 -- 172.21.15.53:0/78544113 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8a38017070 con 0x7f8a48105d40 2024-04-28T12:50:33.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.338+0000 7f8a3e7fc640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8a340a3de0 0x7f8a340a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.338+0000 7f8a4d4a6640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8a340a3de0 0x7f8a340a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.338+0000 7f8a3e7fc640 1 -- 172.21.15.53:0/78544113 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f8a380d09a0 con 0x7f8a48105d40 2024-04-28T12:50:33.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.339+0000 7f8a4d4a6640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8a340a3de0 0x7f8a340a62a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8a44000c00 tx=0x7f8a440314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:33.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.342+0000 7f8a3e7fc640 1 -- 172.21.15.53:0/78544113 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8a3801e030 con 0x7f8a48105d40 2024-04-28T12:50:33.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.527+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a4802dcd0 con 0x7f8a48105d40 2024-04-28T12:50:33.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.529+0000 7f8a3e7fc640 1 -- 172.21.15.53:0/78544113 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7f8a3809d1e0 con 0x7f8a48105d40 2024-04-28T12:50:33.526 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:33.526 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8a340a3de0 msgr2=0x7f8a340a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8a340a3de0 0x7f8a340a62a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8a44000c00 tx=0x7f8a440314a0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48105d40 msgr2=0x7f8a481013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48105d40 0x7f8a481013c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800c970 tx=0x7f8a3800ce30 comp rx=0 tx=0).stop 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 shutdown_connections 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f8a340a3de0 0x7f8a340a62a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a48108670 0x7f8a48101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a48105d40 0x7f8a481013c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 --2- 172.21.15.53:0/78544113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a48103410 0x7f8a48102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 >> 172.21.15.53:0/78544113 conn(0x7f8a480fce20 msgr2=0x7f8a48103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 shutdown_connections 2024-04-28T12:50:33.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.531+0000 7f8a4f70a640 1 -- 172.21.15.53:0/78544113 wait complete. 2024-04-28T12:50:33.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:42.284772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:42.832857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:50:33.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 -- 172.21.15.53:0/912579911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8108650 msgr2=0x7f48d810aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 --2- 172.21.15.53:0/912579911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8108650 0x7f48d810aaa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009900 tx=0x7f48c802a410 comp rx=0 tx=0).stop 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 -- 172.21.15.53:0/912579911 shutdown_connections 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 --2- 172.21.15.53:0/912579911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8108650 0x7f48d810aaa0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 --2- 172.21.15.53:0/912579911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d8105d20 0x7f48d8108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 --2- 172.21.15.53:0/912579911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d81033f0 0x7f48d81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 -- 172.21.15.53:0/912579911 >> 172.21.15.53:0/912579911 conn(0x7f48d80fcde0 msgr2=0x7f48d80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 -- 172.21.15.53:0/912579911 shutdown_connections 2024-04-28T12:50:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.647+0000 7f48deec4640 1 -- 172.21.15.53:0/912579911 wait complete. 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 Processor -- start 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 -- start start 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d81033f0 0x7f48d8102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8105d20 0x7f48d81013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d8108650 0x7f48d81018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f48d810e4a0 con 0x7f48d8105d20 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48d810e320 con 0x7f48d81033f0 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48deec4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f48d810e620 con 0x7f48d8108650 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8105d20 0x7f48d81013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8105d20 0x7f48d81013a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41330/0 (socket says 172.21.15.53:41330) 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 -- 172.21.15.53:0/1888544042 learned_addr learned my addr 172.21.15.53:0/1888544042 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48dcc60640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d81033f0 0x7f48d8102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 -- 172.21.15.53:0/1888544042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d8108650 msgr2=0x7f48d81018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48dd461640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d8108650 0x7f48d81018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d8108650 0x7f48d81018e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 -- 172.21.15.53:0/1888544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d81033f0 msgr2=0x7f48d8102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d81033f0 0x7f48d8102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.648+0000 7f48d7fff640 1 -- 172.21.15.53:0/1888544042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d8102170 con 0x7f48d8105d20 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.649+0000 7f48dcc60640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d81033f0 0x7f48d8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.649+0000 7f48d7fff640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8105d20 0x7f48d81013a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f48c0005fd0 tx=0x7f48c000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.649+0000 7f48d5ffb640 1 -- 172.21.15.53:0/1888544042 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0017070 con 0x7f48d8105d20 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.649+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d81a2c60 con 0x7f48d8105d20 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.649+0000 7f48d5ffb640 1 -- 172.21.15.53:0/1888544042 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48c0009c30 con 0x7f48d8105d20 2024-04-28T12:50:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.649+0000 7f48d5ffb640 1 -- 172.21.15.53:0/1888544042 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0003d30 con 0x7f48d8105d20 2024-04-28T12:50:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.649+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d81a31b0 con 0x7f48d8105d20 2024-04-28T12:50:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.650+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d80fe8d0 con 0x7f48d8105d20 2024-04-28T12:50:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.651+0000 7f48d5ffb640 1 -- 172.21.15.53:0/1888544042 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f48c0003ed0 con 0x7f48d8105d20 2024-04-28T12:50:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.652+0000 7f48d5ffb640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f48c40a3c40 0x7f48c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.652+0000 7f48dcc60640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f48c40a3c40 0x7f48c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.652+0000 7f48d5ffb640 1 -- 172.21.15.53:0/1888544042 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f48c00d0260 con 0x7f48d8105d20 2024-04-28T12:50:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.652+0000 7f48dcc60640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f48c40a3c40 0x7f48c40a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f48d0008e00 tx=0x7f48d0008350 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.656+0000 7f48d5ffb640 1 -- 172.21.15.53:0/1888544042 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f48c0020030 con 0x7f48d8105d20 2024-04-28T12:50:33.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.841+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f48d802dd30 con 0x7f48d8105d20 2024-04-28T12:50:33.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.841+0000 7f48d5ffb640 1 -- 172.21.15.53:0/1888544042 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f48c0009de0 con 0x7f48d8105d20 2024-04-28T12:50:33.838 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f48c40a3c40 msgr2=0x7f48c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f48c40a3c40 0x7f48c40a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f48d0008e00 tx=0x7f48d0008350 comp rx=0 tx=0).stop 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8105d20 msgr2=0x7f48d81013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8105d20 0x7f48d81013a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f48c0005fd0 tx=0x7f48c000dde0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 shutdown_connections 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f48c40a3c40 0x7f48c40a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d8108650 0x7f48d81018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8105d20 0x7f48d81013a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 --2- 172.21.15.53:0/1888544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d81033f0 0x7f48d8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 >> 172.21.15.53:0/1888544042 conn(0x7f48d80fcde0 msgr2=0x7f48d8103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 shutdown_connections 2024-04-28T12:50:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.843+0000 7f48deec4640 1 -- 172.21.15.53:0/1888544042 wait complete. 2024-04-28T12:50:33.850 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:33 smithi053 ceph-mon[29182]: pgmap v347: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:33.850 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:33 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2820073636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:33.850 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:33 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1662814063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:33.850 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:33 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/78544113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:33.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 -- 172.21.15.53:0/2522319680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0108670 msgr2=0x7fbbf010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/2522319680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0108670 0x7fbbf010aac0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009ca0 tx=0x7fbbe802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 -- 172.21.15.53:0/2522319680 shutdown_connections 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/2522319680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0108670 0x7fbbf010aac0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/2522319680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0105d40 0x7fbbf0108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/2522319680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbf0103410 0x7fbbf0105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 -- 172.21.15.53:0/2522319680 >> 172.21.15.53:0/2522319680 conn(0x7fbbf00fce20 msgr2=0x7fbbf00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 -- 172.21.15.53:0/2522319680 shutdown_connections 2024-04-28T12:50:33.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.959+0000 7fbbf7aea640 1 -- 172.21.15.53:0/2522319680 wait complete. 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 Processor -- start 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 -- start start 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbf0103410 0x7fbbf0102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0105d40 0x7fbbf01013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0108670 0x7fbbf0101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf010e530 con 0x7fbbf0105d40 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf010e3b0 con 0x7fbbf0108670 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf7aea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf010e6b0 con 0x7fbbf0103410 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0105d40 0x7fbbf01013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0105d40 0x7fbbf01013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41352/0 (socket says 172.21.15.53:41352) 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 -- 172.21.15.53:0/4142968162 learned_addr learned my addr 172.21.15.53:0/4142968162 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:33.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf6087640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0108670 0x7fbbf0101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5886640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbf0103410 0x7fbbf0102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 -- 172.21.15.53:0/4142968162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbf0103410 msgr2=0x7fbbf0102d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbf0103410 0x7fbbf0102d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 -- 172.21.15.53:0/4142968162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0108670 msgr2=0x7fbbf0101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0108670 0x7fbbf0101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5085640 1 -- 172.21.15.53:0/4142968162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf0102190 con 0x7fbbf0105d40 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf5886640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbf0103410 0x7fbbf0102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.960+0000 7fbbf6087640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0108670 0x7fbbf0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.961+0000 7fbbf5085640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0105d40 0x7fbbf01013c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0005fd0 tx=0x7fbbe000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.961+0000 7fbbe6ffd640 1 -- 172.21.15.53:0/4142968162 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0017070 con 0x7fbbf0105d40 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.961+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf01a2cd0 con 0x7fbbf0105d40 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.961+0000 7fbbe6ffd640 1 -- 172.21.15.53:0/4142968162 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbe0009c30 con 0x7fbbf0105d40 2024-04-28T12:50:33.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.961+0000 7fbbe6ffd640 1 -- 172.21.15.53:0/4142968162 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0003d30 con 0x7fbbf0105d40 2024-04-28T12:50:33.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.961+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf01a31d0 con 0x7fbbf0105d40 2024-04-28T12:50:33.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.962+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf00fe910 con 0x7fbbf0105d40 2024-04-28T12:50:33.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.963+0000 7fbbe6ffd640 1 -- 172.21.15.53:0/4142968162 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbbe0003ed0 con 0x7fbbf0105d40 2024-04-28T12:50:33.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.963+0000 7fbbe6ffd640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbbdc0a3c40 0x7fbbdc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:33.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.963+0000 7fbbf5886640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbbdc0a3c40 0x7fbbdc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:33.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.964+0000 7fbbe6ffd640 1 -- 172.21.15.53:0/4142968162 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fbbe00d0260 con 0x7fbbf0105d40 2024-04-28T12:50:33.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.964+0000 7fbbf5886640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbbdc0a3c40 0x7fbbdc0a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0058c0 tx=0x7fbbec005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:33.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:33.968+0000 7fbbe6ffd640 1 -- 172.21.15.53:0/4142968162 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbbe0020030 con 0x7fbbf0105d40 2024-04-28T12:50:34.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:33 smithi191 ceph-mon[33016]: pgmap v347: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:34.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:33 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2820073636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:34.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:33 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1662814063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:34.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:33 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/78544113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:34.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.157+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbbf0063530 con 0x7fbbf0105d40 2024-04-28T12:50:34.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.158+0000 7fbbe6ffd640 1 -- 172.21.15.53:0/4142968162 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fbbe0009de0 con 0x7fbbf0105d40 2024-04-28T12:50:34.154 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-04-28T12:50:34.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbbdc0a3c40 msgr2=0x7fbbdc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbbdc0a3c40 0x7fbbdc0a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0058c0 tx=0x7fbbec005850 comp rx=0 tx=0).stop 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0105d40 msgr2=0x7fbbf01013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0105d40 0x7fbbf01013c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0005fd0 tx=0x7fbbe000dde0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 shutdown_connections 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbbdc0a3c40 0x7fbbdc0a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0108670 0x7fbbf0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbf0105d40 0x7fbbf01013c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 --2- 172.21.15.53:0/4142968162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbf0103410 0x7fbbf0102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 >> 172.21.15.53:0/4142968162 conn(0x7fbbf00fce20 msgr2=0x7fbbf0103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 shutdown_connections 2024-04-28T12:50:34.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.160+0000 7fbbf7aea640 1 -- 172.21.15.53:0/4142968162 wait complete. 2024-04-28T12:50:34.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:50:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:33 smithi077 ceph-mon[32553]: pgmap v347: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:33 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2820073636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:33 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1662814063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:34.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:33 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/78544113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 -- 172.21.15.53:0/993252803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 msgr2=0x7fbe00108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/993252803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00108110 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009900 tx=0x7fbdf002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 -- 172.21.15.53:0/993252803 shutdown_connections 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/993252803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe00108650 0x7fbe0010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/993252803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00108110 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/993252803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe001033f0 0x7fbe001057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 -- 172.21.15.53:0/993252803 >> 172.21.15.53:0/993252803 conn(0x7fbe000fcde0 msgr2=0x7fbe000ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 -- 172.21.15.53:0/993252803 shutdown_connections 2024-04-28T12:50:34.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.278+0000 7fbe05ca9640 1 -- 172.21.15.53:0/993252803 wait complete. 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 Processor -- start 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 -- start start 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe001033f0 0x7fbe00198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe00108650 0x7fbe001a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010e4e0 con 0x7fbe00105d20 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010e360 con 0x7fbe001033f0 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbe05ca9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010e660 con 0x7fbe00108650 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbdfeffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbdfeffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41374/0 (socket says 172.21.15.53:41374) 2024-04-28T12:50:34.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbdfeffd640 1 -- 172.21.15.53:0/960917804 learned_addr learned my addr 172.21.15.53:0/960917804 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.279+0000 7fbdff7fe640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe001033f0 0x7fbe00198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfffff640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe00108650 0x7fbe001a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfeffd640 1 -- 172.21.15.53:0/960917804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe00108650 msgr2=0x7fbe001a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfeffd640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe00108650 0x7fbe001a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfeffd640 1 -- 172.21.15.53:0/960917804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe001033f0 msgr2=0x7fbe00198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfeffd640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe001033f0 0x7fbe00198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfeffd640 1 -- 172.21.15.53:0/960917804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe001a0740 con 0x7fbe00105d20 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfffff640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe00108650 0x7fbe001a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfeffd640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00199060 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0031500 tx=0x7fbdf0003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfcff9640 1 -- 172.21.15.53:0/960917804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf003f070 con 0x7fbe00105d20 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfcff9640 1 -- 172.21.15.53:0/960917804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdf003b460 con 0x7fbe00105d20 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe001a09d0 con 0x7fbe00105d20 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbdfcff9640 1 -- 172.21.15.53:0/960917804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0035030 con 0x7fbe00105d20 2024-04-28T12:50:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.280+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe001a0ec0 con 0x7fbe00105d20 2024-04-28T12:50:34.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.282+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe000fe8d0 con 0x7fbe00105d20 2024-04-28T12:50:34.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.282+0000 7fbdfcff9640 1 -- 172.21.15.53:0/960917804 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbdf0042070 con 0x7fbe00105d20 2024-04-28T12:50:34.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.283+0000 7fbdfcff9640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbde80a3c40 0x7fbde80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.283+0000 7fbdff7fe640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbde80a3c40 0x7fbde80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.283+0000 7fbdfcff9640 1 -- 172.21.15.53:0/960917804 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fbdf00f0f00 con 0x7fbe00105d20 2024-04-28T12:50:34.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.283+0000 7fbdff7fe640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbde80a3c40 0x7fbde80a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400abe0 tx=0x7fbdf400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:34.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.287+0000 7fbdfcff9640 1 -- 172.21.15.53:0/960917804 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbdf00bd780 con 0x7fbe00105d20 2024-04-28T12:50:34.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.486+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbe0002dcd0 con 0x7fbe00105d20 2024-04-28T12:50:34.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.487+0000 7fbdfcff9640 1 -- 172.21.15.53:0/960917804 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fbdf00bd080 con 0x7fbe00105d20 2024-04-28T12:50:34.484 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbde80a3c40 msgr2=0x7fbde80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbde80a3c40 0x7fbde80a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400abe0 tx=0x7fbdf400a2d0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 msgr2=0x7fbe00199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00199060 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0031500 tx=0x7fbdf0003d50 comp rx=0 tx=0).stop 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 shutdown_connections 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbde80a3c40 0x7fbde80a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe00108650 0x7fbe001a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe00105d20 0x7fbe00199060 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 --2- 172.21.15.53:0/960917804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe001033f0 0x7fbe00198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 >> 172.21.15.53:0/960917804 conn(0x7fbe000fcde0 msgr2=0x7fbe00103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:34.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 shutdown_connections 2024-04-28T12:50:34.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.489+0000 7fbe05ca9640 1 -- 172.21.15.53:0/960917804 wait complete. 2024-04-28T12:50:34.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:34.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- 172.21.15.53:0/4066767144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950105d80 msgr2=0x7f9950108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 --2- 172.21.15.53:0/4066767144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950105d80 0x7f9950108170 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9940009900 tx=0x7f994002a410 comp rx=0 tx=0).stop 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- 172.21.15.53:0/4066767144 shutdown_connections 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 --2- 172.21.15.53:0/4066767144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99501086b0 0x7f995010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 --2- 172.21.15.53:0/4066767144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950105d80 0x7f9950108170 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 --2- 172.21.15.53:0/4066767144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9950103450 0x7f9950105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- 172.21.15.53:0/4066767144 >> 172.21.15.53:0/4066767144 conn(0x7f99500fcde0 msgr2=0x7f99500ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- 172.21.15.53:0/4066767144 shutdown_connections 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- 172.21.15.53:0/4066767144 wait complete. 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 Processor -- start 2024-04-28T12:50:34.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- start start 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950103450 0x7f9950198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9950105d80 0x7f9950199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99501086b0 0x7f99501a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e4a0 con 0x7f9950103450 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.606+0000 7f995841d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e320 con 0x7f9950105d80 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f995841d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e620 con 0x7f99501086b0 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950103450 0x7f9950198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950103450 0x7f9950198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41386/0 (socket says 172.21.15.53:41386) 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 -- 172.21.15.53:0/3914082184 learned_addr learned my addr 172.21.15.53:0/3914082184 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99559b8640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9950105d80 0x7f9950199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99569ba640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99501086b0 0x7f99501a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 -- 172.21.15.53:0/3914082184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99501086b0 msgr2=0x7f99501a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99501086b0 0x7f99501a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 -- 172.21.15.53:0/3914082184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9950105d80 msgr2=0x7f9950199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9950105d80 0x7f9950199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 -- 172.21.15.53:0/3914082184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99501a0810 con 0x7f9950103450 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99561b9640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950103450 0x7f9950198c40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f994c00eab0 tx=0x7f994c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99477fe640 1 -- 172.21.15.53:0/3914082184 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c00ce40 con 0x7f9950103450 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99477fe640 1 -- 172.21.15.53:0/3914082184 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f994c011920 con 0x7f9950103450 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99501a0b00 con 0x7f9950103450 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f99477fe640 1 -- 172.21.15.53:0/3914082184 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c010a30 con 0x7f9950103450 2024-04-28T12:50:34.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.607+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99501a1020 con 0x7f9950103450 2024-04-28T12:50:34.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.609+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99500fe8d0 con 0x7f9950103450 2024-04-28T12:50:34.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.610+0000 7f99477fe640 1 -- 172.21.15.53:0/3914082184 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f994c010bd0 con 0x7f9950103450 2024-04-28T12:50:34.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.611+0000 7f99477fe640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f993c0a3c40 0x7f993c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.611+0000 7f99559b8640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f993c0a3c40 0x7f993c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.611+0000 7f99477fe640 1 -- 172.21.15.53:0/3914082184 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f994c0d0b90 con 0x7f9950103450 2024-04-28T12:50:34.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.611+0000 7f99559b8640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f993c0a3c40 0x7f993c0a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9940009230 tx=0x7f99400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:34.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.615+0000 7f99477fe640 1 -- 172.21.15.53:0/3914082184 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f994c016030 con 0x7f9950103450 2024-04-28T12:50:34.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.842+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f995002dad0 con 0x7f9950103450 2024-04-28T12:50:34.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.843+0000 7f99477fe640 1 -- 172.21.15.53:0/3914082184 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11873 (secure 0 0 0) 0x7f995002dad0 con 0x7f9950103450 2024-04-28T12:50:34.840 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:34.840 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:48:49.455677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24485,24478]},"id":1}]} 2024-04-28T12:50:34.840 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 22 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f993c0a3c40 msgr2=0x7f993c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f993c0a3c40 0x7f993c0a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9940009230 tx=0x7f99400314a0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950103450 msgr2=0x7f9950198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950103450 0x7f9950198c40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f994c00eab0 tx=0x7f994c00ef70 comp rx=0 tx=0).stop 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 shutdown_connections 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f993c0a3c40 0x7f993c0a6100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99501086b0 0x7f99501a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9950105d80 0x7f9950199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 --2- 172.21.15.53:0/3914082184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9950103450 0x7f9950198c40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 >> 172.21.15.53:0/3914082184 conn(0x7f99500fcde0 msgr2=0x7f9950103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 shutdown_connections 2024-04-28T12:50:34.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.845+0000 7f995841d640 1 -- 172.21.15.53:0/3914082184 wait complete. 2024-04-28T12:50:34.853 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:34.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 -- 172.21.15.53:0/2934313391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4103440 msgr2=0x7f42d4109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 --2- 172.21.15.53:0/2934313391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4103440 0x7f42d4109cd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009ca0 tx=0x7f42cc02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 -- 172.21.15.53:0/2934313391 shutdown_connections 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 --2- 172.21.15.53:0/2934313391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4103440 0x7f42d4109cd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 --2- 172.21.15.53:0/2934313391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42d4102a80 0x7f42d4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 --2- 172.21.15.53:0/2934313391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d4101890 0x7f42d4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 -- 172.21.15.53:0/2934313391 >> 172.21.15.53:0/2934313391 conn(0x7f42d40fce20 msgr2=0x7f42d40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 -- 172.21.15.53:0/2934313391 shutdown_connections 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 -- 172.21.15.53:0/2934313391 wait complete. 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.960+0000 7f42d99e4640 1 Processor -- start 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d99e4640 1 -- start start 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d99e4640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4101890 0x7f42d4198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d99e4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42d4102a80 0x7f42d41990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d99e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d4103440 0x7f42d41a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d99e4640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f42d410d3c0 con 0x7f42d4101890 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d99e4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42d410d240 con 0x7f42d4103440 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d99e4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f42d410d540 con 0x7f42d4102a80 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4101890 0x7f42d4198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4101890 0x7f42d4198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41412/0 (socket says 172.21.15.53:41412) 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 -- 172.21.15.53:0/4162936264 learned_addr learned my addr 172.21.15.53:0/4162936264 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d37fe640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d4103440 0x7f42d41a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d27fc640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42d4102a80 0x7f42d41990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 -- 172.21.15.53:0/4162936264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42d4102a80 msgr2=0x7f42d41990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42d4102a80 0x7f42d41990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 -- 172.21.15.53:0/4162936264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d4103440 msgr2=0x7f42d41a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:34.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d4103440 0x7f42d41a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 -- 172.21.15.53:0/4162936264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d41a07b0 con 0x7f42d4101890 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d37fe640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d4103440 0x7f42d41a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.961+0000 7f42d2ffd640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4101890 0x7f42d4198b90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c970 tx=0x7f42c400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.962+0000 7f42d89e2640 1 -- 172.21.15.53:0/4162936264 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4004220 con 0x7f42d4101890 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.962+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d41a0aa0 con 0x7f42d4101890 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.962+0000 7f42d89e2640 1 -- 172.21.15.53:0/4162936264 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42c40043c0 con 0x7f42d4101890 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.962+0000 7f42d89e2640 1 -- 172.21.15.53:0/4162936264 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c400f450 con 0x7f42d4101890 2024-04-28T12:50:34.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.962+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d41a0ff0 con 0x7f42d4101890 2024-04-28T12:50:34.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.963+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d41a1280 con 0x7f42d4101890 2024-04-28T12:50:34.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.964+0000 7f42d89e2640 1 -- 172.21.15.53:0/4162936264 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f42c4003610 con 0x7f42d4101890 2024-04-28T12:50:34.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.964+0000 7f42d89e2640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42bc0a3c40 0x7f42bc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:34.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.964+0000 7f42d27fc640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42bc0a3c40 0x7f42bc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:34.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.964+0000 7f42d89e2640 1 -- 172.21.15.53:0/4162936264 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f42c40d04c0 con 0x7f42d4101890 2024-04-28T12:50:34.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.965+0000 7f42d27fc640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42bc0a3c40 0x7f42bc0a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f42c8005fd0 tx=0x7f42c8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:34.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:34.969+0000 7f42d89e2640 1 -- 172.21.15.53:0/4162936264 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f42c4014030 con 0x7f42d4101890 2024-04-28T12:50:35.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:34 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1888544042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:34 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4142968162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:34 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/960917804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:34 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1888544042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:34 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4142968162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:34 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/960917804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.157+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42d41a1450 con 0x7f42d4101890 2024-04-28T12:50:35.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.159+0000 7f42d89e2640 1 -- 172.21.15.53:0/4162936264 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7f42c4098030 con 0x7f42d4101890 2024-04-28T12:50:35.156 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T12:50:35.156 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42bc0a3c40 msgr2=0x7f42bc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42bc0a3c40 0x7f42bc0a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f42c8005fd0 tx=0x7f42c8005850 comp rx=0 tx=0).stop 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4101890 msgr2=0x7f42d4198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4101890 0x7f42d4198b90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c970 tx=0x7f42c400ce30 comp rx=0 tx=0).stop 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 shutdown_connections 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f42bc0a3c40 0x7f42bc0a6100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d4103440 0x7f42d41a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42d4102a80 0x7f42d41990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 --2- 172.21.15.53:0/4162936264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42d4101890 0x7f42d4198b90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 >> 172.21.15.53:0/4162936264 conn(0x7f42d40fce20 msgr2=0x7f42d40fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 shutdown_connections 2024-04-28T12:50:35.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.161+0000 7f42d99e4640 1 -- 172.21.15.53:0/4162936264 wait complete. 2024-04-28T12:50:35.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-28T12:48:43.873799+0000', 'modified': '2024-04-28T12:48:49.455677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24505, 'mds_1': 24485, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6833/3944921810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 3944921810}, {'type': 'v1', 'addr': '172.21.15.77:6833', 'nonce': 3944921810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6835/1670611904', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 1670611904}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 1670611904}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/2819920278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 2819920278}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 2819920278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.191:6837/4132012864', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 4132012864}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 4132012864}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.77:6837/2147382100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2147382100}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2147382100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6839/2399731075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 2399731075}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 2399731075}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24505, 'qdb_cluster': [24505, 24485, 24478]} 2024-04-28T12:50:35.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-28T12:50:35.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-28T12:50:35.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:34 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1888544042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:34 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4142968162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:34 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/960917804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:35.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 -- 172.21.15.53:0/3624208405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 msgr2=0x7fa86c10aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 --2- 172.21.15.53:0/3624208405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c10aaa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa864009ca0 tx=0x7fa86402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 -- 172.21.15.53:0/3624208405 shutdown_connections 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 --2- 172.21.15.53:0/3624208405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c10aaa0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 --2- 172.21.15.53:0/3624208405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa86c105d20 0x7fa86c108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 --2- 172.21.15.53:0/3624208405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c1033f0 0x7fa86c1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 -- 172.21.15.53:0/3624208405 >> 172.21.15.53:0/3624208405 conn(0x7fa86c0fcde0 msgr2=0x7fa86c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 -- 172.21.15.53:0/3624208405 shutdown_connections 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 -- 172.21.15.53:0/3624208405 wait complete. 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 Processor -- start 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.277+0000 7fa874732640 1 -- start start 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c1033f0 0x7fa86c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa86c105d20 0x7fa86c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c10e4c0 con 0x7fa86c108650 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c10e340 con 0x7fa86c1033f0 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c10e640 con 0x7fa86c105d20 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c1a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41428/0 (socket says 172.21.15.53:41428) 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 -- 172.21.15.53:0/2733178078 learned_addr learned my addr 172.21.15.53:0/2733178078 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa8724ce640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c1033f0 0x7fa86c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa871ccd640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa86c105d20 0x7fa86c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 -- 172.21.15.53:0/2733178078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa86c105d20 msgr2=0x7fa86c199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa86c105d20 0x7fa86c199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 -- 172.21.15.53:0/2733178078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c1033f0 msgr2=0x7fa86c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c1033f0 0x7fa86c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 -- 172.21.15.53:0/2733178078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c1a0810 con 0x7fa86c108650 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa872ccf640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c1a01a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa8640097b0 tx=0x7fa864003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa85f7fe640 1 -- 172.21.15.53:0/2733178078 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864004240 con 0x7fa86c108650 2024-04-28T12:50:35.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa85f7fe640 1 -- 172.21.15.53:0/2733178078 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8640317c0 con 0x7fa86c108650 2024-04-28T12:50:35.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86c1a0aa0 con 0x7fa86c108650 2024-04-28T12:50:35.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa85f7fe640 1 -- 172.21.15.53:0/2733178078 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86402fd80 con 0x7fa86c108650 2024-04-28T12:50:35.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.278+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa86c1a1040 con 0x7fa86c108650 2024-04-28T12:50:35.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.281+0000 7fa87744d740 1 Processor -- start 2024-04-28T12:50:35.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.281+0000 7fa85f7fe640 1 -- 172.21.15.53:0/2733178078 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa8640043e0 con 0x7fa86c108650 2024-04-28T12:50:35.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa87744d740 1 -- start start 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa87744d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c0dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa87744d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0e080 0x562861c12b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa87744d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c15680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa87744d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562861b72740 con 0x562861c0d690 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa87744d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562861b0e3a0 con 0x562861c131f0 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa87744d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562861b6e700 con 0x562861c0e080 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c0dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c0dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41440/0 (socket says 172.21.15.53:41440) 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 -- 172.21.15.53:0/4197495038 learned_addr learned my addr 172.21.15.53:0/4197495038 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa8724ce640 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0e080 0x562861c12b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c15680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa85f7fe640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa8580a3cc0 0x7fa8580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 -- 172.21.15.53:0/4197495038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0e080 msgr2=0x562861c12b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0e080 0x562861c12b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 -- 172.21.15.53:0/4197495038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 msgr2=0x562861c15680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c15680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa871ccd640 1 -- 172.21.15.53:0/4197495038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562861b2a750 con 0x562861c0d690 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.282+0000 7fa872ccf640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa8580a3cc0 0x7fa8580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.283+0000 7fa85f7fe640 1 -- 172.21.15.53:0/2733178078 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fa8640f1c40 con 0x7fa86c108650 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.283+0000 7fa871ccd640 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c0dab0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa86800bbf0 tx=0x7fa868004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157c6ec1e2a5a1c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.283+0000 7fa85cff9640 1 -- 172.21.15.53:0/4197495038 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868005020 con 0x562861c0d690 2024-04-28T12:50:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.283+0000 7fa85cff9640 1 -- 172.21.15.53:0/4197495038 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868005670 con 0x562861c0d690 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.283+0000 7fa85cff9640 1 -- 172.21.15.53:0/4197495038 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868010460 con 0x562861c0d690 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.283+0000 7fa87744d740 1 -- 172.21.15.53:0/4197495038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 msgr2=0x562861c0dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c0dab0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa86800bbf0 tx=0x7fa868004610 comp rx=0 tx=0).stop 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.283+0000 7fa872ccf640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa8580a3cc0 0x7fa8580a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa86404e000 tx=0x7fa86402a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- 172.21.15.53:0/4197495038 shutdown_connections 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c15680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0e080 0x562861c12b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 --2- 172.21.15.53:0/4197495038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c0dab0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- 172.21.15.53:0/4197495038 >> 172.21.15.53:0/4197495038 conn(0x562861c08d00 msgr2=0x562861c09100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- 172.21.15.53:0/4197495038 shutdown_connections 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- 172.21.15.53:0/4197495038 wait complete. 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- start start 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 Processor -- start 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c17ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c17ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c131f0 0x562861c1f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562861c16140 con 0x562861c131f0 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562861af9fe0 con 0x562861c0e080 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562861c162c0 con 0x562861c0d690 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa87744d740 10 client.0 _ll_register_callbacks cb 0x562861b10150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa871ccd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c131f0 0x562861c1f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa871ccd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c131f0 0x562861c1f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41452/0 (socket says 172.21.15.53:41452) 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa871ccd640 1 -- 172.21.15.53:0/619301422 learned_addr learned my addr 172.21.15.53:0/619301422 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.284+0000 7fa8724ce640 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c17ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa871ccd640 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c17ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa871ccd640 1 -- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 msgr2=0x562861c17ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa871ccd640 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c17ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa871ccd640 1 -- 172.21.15.53:0/619301422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 msgr2=0x562861c17ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa871ccd640 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c17ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa871ccd640 1 -- 172.21.15.53:0/619301422 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628618842d0 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa871ccd640 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c131f0 0x562861c1f030 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa868004f90 tx=0x7fa86800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cb87bc354b1755 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa855ffb640 1 -- 172.21.15.53:0/619301422 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8680051c0 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa855ffb640 1 -- 172.21.15.53:0/619301422 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fa868005360 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa868005670 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa855ffb640 1 -- 172.21.15.53:0/619301422 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa868010460 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa87744d740 10 client.15216 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56286194e6b0 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.285+0000 7fa855ffb640 1 -- 172.21.15.53:0/619301422 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868005670 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa855ffb640 1 -- 172.21.15.53:0/619301422 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa868020500 con 0x562861c131f0 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa87744d740 10 client.15216 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa87744d740 10 client.15216 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-28T12:50:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa87744d740 20 client.15216 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa87744d740 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c25710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa87744d740 4 client.15216 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 --> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562861b80350 con 0x562861c242a0 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa8724ce640 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c25710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa855ffb640 10 client.15216 ms_handle_connect on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.286+0000 7fa8724ce640 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c25710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f19ab8ac033d2fae server_cookie=ba179982de6334f1 in_seq=0 out_seq=0 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa855ffb640 1 -- 172.21.15.53:0/619301422 <== mds.2 v2:172.21.15.191:6836/4132012864 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562861b80350 con 0x562861c242a0 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa855ffb640 10 client.15216 handle_command_reply: tid=0 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa87744d740 1 client.15216 shutdown 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c131f0 msgr2=0x562861c1f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa87744d740 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c131f0 0x562861c1f030 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa868004f90 tx=0x7fa86800b5b0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 shutdown_connections 2024-04-28T12:50:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa87744d740 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c131f0 0x562861c1f030 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa87744d740 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c17ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.287+0000 7fa87744d740 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c25710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.288+0000 7fa87744d740 1 --2- 172.21.15.53:0/619301422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c17ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.288+0000 7fa855ffb640 0 client.15216 ms_handle_reset on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.288+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 >> 172.21.15.53:0/619301422 conn(0x562861c08d00 msgr2=0x562861c07f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.288+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 shutdown_connections 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.288+0000 7fa87744d740 1 -- 172.21.15.53:0/619301422 wait complete. 2024-04-28T12:50:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.288+0000 7fa87744d740 20 client.15216 trim_cache size 0 max 16384 2024-04-28T12:50:35.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.323+0000 7fa87744d740 1 Processor -- start 2024-04-28T12:50:35.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.323+0000 7fa87744d740 1 -- start start 2024-04-28T12:50:35.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa87744d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c17f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa87744d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c1ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa87744d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c131f0 0x562861c07670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa87744d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562861c23a50 con 0x562861c0d690 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa87744d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562861c24120 con 0x562861c0e080 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa87744d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562861c247f0 con 0x562861c131f0 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c17f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c17f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41454/0 (socket says 172.21.15.53:41454) 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 -- 172.21.15.53:0/129991080 learned_addr learned my addr 172.21.15.53:0/129991080 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 -- 172.21.15.53:0/129991080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c131f0 msgr2=0x562861c07670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa8724ce640 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c1ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c131f0 0x562861c07670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 -- 172.21.15.53:0/129991080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 msgr2=0x562861c1ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c1ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 -- 172.21.15.53:0/129991080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562861c088a0 con 0x562861c0d690 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa871ccd640 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c17f50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa868000c00 tx=0x7fa86800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a56055a41d53d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa857fff640 1 -- 172.21.15.53:0/129991080 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868004a50 con 0x562861c0d690 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa857fff640 1 -- 172.21.15.53:0/129991080 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868004bf0 con 0x562861c0d690 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.324+0000 7fa857fff640 1 -- 172.21.15.53:0/129991080 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868005340 con 0x562861c0d690 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 -- 172.21.15.53:0/129991080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 msgr2=0x562861c17f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c17f50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa868000c00 tx=0x7fa86800bda0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 -- 172.21.15.53:0/129991080 shutdown_connections 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c131f0 0x562861c07670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c0e080 0x562861c1ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 --2- 172.21.15.53:0/129991080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0d690 0x562861c17f50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 -- 172.21.15.53:0/129991080 >> 172.21.15.53:0/129991080 conn(0x562861c08d00 msgr2=0x562861c090e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 -- 172.21.15.53:0/129991080 shutdown_connections 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.325+0000 7fa87744d740 1 -- 172.21.15.53:0/129991080 wait complete. 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 -- start start 2024-04-28T12:50:35.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 Processor -- start 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c1b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0e080 0x562861c1b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c1e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x562861c25590 con 0x562861c0e080 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562861c24ec0 con 0x562861c131f0 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562861c25c60 con 0x562861c0d690 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa87744d740 10 client.0 _ll_register_callbacks cb 0x7fa86000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0e080 0x562861c1b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0e080 0x562861c1b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41466/0 (socket says 172.21.15.53:41466) 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 -- 172.21.15.53:0/2379097776 learned_addr learned my addr 172.21.15.53:0/2379097776 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa871ccd640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c1b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa871ccd640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c1e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 -- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 msgr2=0x562861c1b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c1b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 -- 172.21.15.53:0/2379097776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 msgr2=0x562861c1e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c1e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 -- 172.21.15.53:0/2379097776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562861c09a70 con 0x562861c0e080 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa871ccd640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c1e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa8724ce640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0e080 0x562861c1b6c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa860000c00 tx=0x7fa86000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35373a47a4ca290d server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.326+0000 7fa855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa855ffb640 1 -- 172.21.15.53:0/2379097776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600200e0 con 0x562861c0e080 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa855ffb640 1 -- 172.21.15.53:0/2379097776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fa860020280 con 0x562861c0e080 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa868004bf0 con 0x562861c0e080 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa855ffb640 1 -- 172.21.15.53:0/2379097776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa860020480 con 0x562861c0e080 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 10 client.15222 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:35.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562861c8abb0 con 0x562861c0e080 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa855ffb640 1 -- 172.21.15.53:0/2379097776 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868004bf0 con 0x562861c0e080 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa855ffb640 1 -- 172.21.15.53:0/2379097776 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa860011c80 con 0x562861c0e080 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 10 client.15222 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 10 client.15222 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 20 client.15222 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c8e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 4 client.15222 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.327+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 --> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] -- command(tid 0: {"prefix": "status"}) -- 0x562861b7a1c0 con 0x562861c242a0 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa871ccd640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c8e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa871ccd640 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c8e540 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e73a13df8ef8dc6c server_cookie=c5459397a0a68b22 in_seq=0 out_seq=0 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa855ffb640 10 client.15222 ms_handle_connect on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa855ffb640 1 -- 172.21.15.53:0/2379097776 <== mds.2 v2:172.21.15.191:6836/4132012864 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562861b7a1c0 con 0x562861c242a0 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa855ffb640 10 client.15222 handle_command_reply: tid=0 2024-04-28T12:50:35.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 client.15222 shutdown 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0e080 msgr2=0x562861c1b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0e080 0x562861c1b6c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa860000c00 tx=0x7fa86000be30 comp rx=0 tx=0).stop 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 shutdown_connections 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562861c131f0 0x562861c1e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562861c0e080 0x562861c1b6c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:6836/4132012864,v1:172.21.15.191:6837/4132012864] conn(0x562861c242a0 0x562861c8e540 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 --2- 172.21.15.53:0/2379097776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562861c0d690 0x562861c1b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa855ffb640 0 client.15222 ms_handle_reset on v2:172.21.15.191:6836/4132012864 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.328+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 >> 172.21.15.53:0/2379097776 conn(0x562861c08d00 msgr2=0x562861c0bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 shutdown_connections 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa87744d740 1 -- 172.21.15.53:0/2379097776 wait complete. 2024-04-28T12:50:35.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa87744d740 20 client.15222 trim_cache size 0 max 16384 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 2, 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24478, 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 106.78661018299999, 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 22, 2024-04-28T12:50:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 88, 2024-04-28T12:50:35.334 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 86, 2024-04-28T12:50:35.334 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 131.30772693599999 2024-04-28T12:50:35.334 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-28T12:50:35.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa8580a3cc0 msgr2=0x7fa8580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa8580a3cc0 0x7fa8580a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa86404e000 tx=0x7fa86402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 msgr2=0x7fa86c1a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c1a01a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa8640097b0 tx=0x7fa864003c20 comp rx=0 tx=0).stop 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 shutdown_connections 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fa8580a3cc0 0x7fa8580a6180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa86c108650 0x7fa86c1a01a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa86c105d20 0x7fa86c199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 --2- 172.21.15.53:0/2733178078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c1033f0 0x7fa86c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 >> 172.21.15.53:0/2733178078 conn(0x7fa86c0fcde0 msgr2=0x7fa86c103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 shutdown_connections 2024-04-28T12:50:35.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.329+0000 7fa874732640 1 -- 172.21.15.53:0/2733178078 wait complete. 2024-04-28T12:50:35.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 -- 172.21.15.53:0/628569846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 msgr2=0x7f0c6c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/628569846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c108110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009900 tx=0x7f0c5c02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 -- 172.21.15.53:0/628569846 shutdown_connections 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/628569846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c6c108650 0x7f0c6c10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/628569846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c108110 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/628569846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6c1033f0 0x7f0c6c1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 -- 172.21.15.53:0/628569846 >> 172.21.15.53:0/628569846 conn(0x7f0c6c0fcde0 msgr2=0x7f0c6c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 -- 172.21.15.53:0/628569846 shutdown_connections 2024-04-28T12:50:35.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.445+0000 7f0c70fc6640 1 -- 172.21.15.53:0/628569846 wait complete. 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 Processor -- start 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 -- start start 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c6c1033f0 0x7f0c6c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6c108650 0x7f0c6c1a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c10e4a0 con 0x7f0c6c105d20 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c10e320 con 0x7f0c6c108650 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c70fc6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c10e620 con 0x7f0c6c1033f0 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c69d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c69d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41484/0 (socket says 172.21.15.53:41484) 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.446+0000 7f0c69d9b640 1 -- 172.21.15.53:0/2246998624 learned_addr learned my addr 172.21.15.53:0/2246998624 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c6c1033f0 0x7f0c6c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6c108650 0x7f0c6c1a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6a59c640 1 -- 172.21.15.53:0/2246998624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6c108650 msgr2=0x7f0c6c1a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6c108650 0x7f0c6c1a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6a59c640 1 -- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 msgr2=0x7f0c6c199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6a59c640 1 -- 172.21.15.53:0/2246998624 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6c1a0860 con 0x7f0c6c1033f0 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c6c1033f0 0x7f0c6c198c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e9f0 tx=0x7f0c6000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c5b7fe640 1 -- 172.21.15.53:0/2246998624 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6000ccc0 con 0x7f0c6c1033f0 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6c1a0b50 con 0x7f0c6c1033f0 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c5b7fe640 1 -- 172.21.15.53:0/2246998624 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c600108e0 con 0x7f0c6c1033f0 2024-04-28T12:50:35.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c5b7fe640 1 -- 172.21.15.53:0/2246998624 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60018880 con 0x7f0c6c1033f0 2024-04-28T12:50:35.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.447+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c6c1a10f0 con 0x7f0c6c1033f0 2024-04-28T12:50:35.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.450+0000 7f0c73dc4740 1 Processor -- start 2024-04-28T12:50:35.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.450+0000 7f0c73dc4740 1 -- start start 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c5b7fe640 1 -- 172.21.15.53:0/2246998624 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0c60010430 con 0x7f0c6c1033f0 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d774aa0 0x55868d774ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d775490 0x55868d779f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d77ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55868d6d5740 con 0x55868d775490 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55868d6713a0 con 0x55868d774aa0 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55868d6d1700 con 0x55868d77a600 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d775490 0x55868d779f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d775490 0x55868d779f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41488/0 (socket says 172.21.15.53:41488) 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3092598 learned_addr learned my addr 172.21.15.53:0/3092598 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d774aa0 0x55868d774ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d77ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3092598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 msgr2=0x55868d77ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d77ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3092598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d774aa0 msgr2=0x55868d774ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d774aa0 0x55868d774ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3092598 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55868d68d750 con 0x55868d775490 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d77ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.451+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d775490 0x55868d779f90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0c600ce200 tx=0x7f0c600d82a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0c78ff7b866bea server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.452+0000 7f0c5b7fe640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0c540a3c40 0x7f0c540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.452+0000 7f0c58ff9640 1 -- 172.21.15.53:0/3092598 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c600df070 con 0x55868d775490 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.452+0000 7f0c58ff9640 1 -- 172.21.15.53:0/3092598 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c6009cb40 con 0x55868d775490 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.452+0000 7f0c58ff9640 1 -- 172.21.15.53:0/3092598 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6009ac80 con 0x55868d775490 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.452+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0c540a3c40 0x7f0c540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.452+0000 7f0c5b7fe640 1 -- 172.21.15.53:0/2246998624 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f0c600d0910 con 0x7f0c6c1033f0 2024-04-28T12:50:35.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3092598 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d775490 msgr2=0x55868d779f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0c540a3c40 0x7f0c540a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55868d775cf0 tx=0x7f0c64009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d775490 0x55868d779f90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0c600ce200 tx=0x7f0c600d82a0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3092598 shutdown_connections 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d77ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d775490 0x55868d779f90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3092598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d774aa0 0x55868d774ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3092598 >> 172.21.15.53:0/3092598 conn(0x55868d76bd00 msgr2=0x55868d76c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3092598 shutdown_connections 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3092598 wait complete. 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- start start 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 Processor -- start 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d77eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d77f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d786440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55868d77d5b0 con 0x55868d774aa0 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55868d65cfe0 con 0x55868d775490 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55868d77d730 con 0x55868d77a600 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d77eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.453+0000 7f0c73dc4740 10 client.0 _ll_register_callbacks cb 0x55868d673150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d77eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41490/0 (socket says 172.21.15.53:41490) 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3715539935 learned_addr learned my addr 172.21.15.53:0/3715539935 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d77f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d786440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3715539935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 msgr2=0x55868d786440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d786440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 msgr2=0x55868d77f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d77f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 -- 172.21.15.53:0/3715539935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55868d3e72d0 con 0x55868d774aa0 2024-04-28T12:50:35.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d77f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d786440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d77eec0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0c6009cad0 tx=0x7f0c600beb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1934d643562cc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c51ffb640 1 -- 172.21.15.53:0/3715539935 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c600bd080 con 0x55868d774aa0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c51ffb640 1 -- 172.21.15.53:0/3715539935 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f0c60010a90 con 0x55868d774aa0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c6009cb40 con 0x55868d774aa0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c51ffb640 1 -- 172.21.15.53:0/3715539935 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0c600bd3d0 con 0x55868d774aa0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c73dc4740 10 client.15234 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.454+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55868d4b16b0 con 0x55868d774aa0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c51ffb640 1 -- 172.21.15.53:0/3715539935 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c6009cb40 con 0x55868d774aa0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c51ffb640 1 -- 172.21.15.53:0/3715539935 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0c60099120 con 0x55868d774aa0 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c73dc4740 10 client.15234 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c73dc4740 10 client.15234 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:50:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c73dc4740 20 client.15234 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:35.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d78b3c0 0x55868d78c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c73dc4740 4 client.15234 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:35.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 --> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55868d6e3350 con 0x55868d78b3c0 2024-04-28T12:50:35.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.455+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d78b3c0 0x55868d78c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.456+0000 7f0c51ffb640 10 client.15234 ms_handle_connect on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:35.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.456+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d78b3c0 0x55868d78c830 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e5e84862525905f server_cookie=d26391c144618ebb in_seq=0 out_seq=0 2024-04-28T12:50:35.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.456+0000 7f0c51ffb640 1 -- 172.21.15.53:0/3715539935 <== mds.1 v2:172.21.15.77:6836/2147382100 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55868d6e3350 con 0x55868d78b3c0 2024-04-28T12:50:35.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.456+0000 7f0c51ffb640 10 client.15234 handle_command_reply: tid=0 2024-04-28T12:50:35.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.456+0000 7f0c73dc4740 1 client.15234 shutdown 2024-04-28T12:50:35.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 msgr2=0x55868d77eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d77eec0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0c6009cad0 tx=0x7f0c600beb50 comp rx=0 tx=0).stop 2024-04-28T12:50:35.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 shutdown_connections 2024-04-28T12:50:35.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d77a600 0x55868d786440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d77f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d78b3c0 0x55868d78c830 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/3715539935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d77eec0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 >> 172.21.15.53:0/3715539935 conn(0x55868d76bd00 msgr2=0x55868d76af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c51ffb640 0 client.15234 ms_handle_reset on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 shutdown_connections 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 1 -- 172.21.15.53:0/3715539935 wait complete. 2024-04-28T12:50:35.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.457+0000 7f0c73dc4740 20 client.15234 trim_cache size 0 max 16384 2024-04-28T12:50:35.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 Processor -- start 2024-04-28T12:50:35.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 -- start start 2024-04-28T12:50:35.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d766fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d775490 0x55868d786400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d77a600 0x55868d76c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55868d78ab70 con 0x55868d774aa0 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55868d78b240 con 0x55868d77a600 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55868d78b910 con 0x55868d775490 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.475+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d766fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d766fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41494/0 (socket says 172.21.15.53:41494) 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 -- 172.21.15.53:0/1389258608 learned_addr learned my addr 172.21.15.53:0/1389258608 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d775490 0x55868d786400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 -- 172.21.15.53:0/1389258608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d775490 msgr2=0x55868d786400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d775490 0x55868d786400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d77a600 0x55868d76c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 -- 172.21.15.53:0/1389258608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d77a600 msgr2=0x55868d76c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d77a600 0x55868d76c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 -- 172.21.15.53:0/1389258608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55868d779b70 con 0x55868d774aa0 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d775490 0x55868d786400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d766fd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0c600be930 tx=0x7f0c600d69e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12e63ca547cef81 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c53fff640 1 -- 172.21.15.53:0/1389258608 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c600df070 con 0x55868d774aa0 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c53fff640 1 -- 172.21.15.53:0/1389258608 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c6009cb40 con 0x55868d774aa0 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.476+0000 7f0c53fff640 1 -- 172.21.15.53:0/1389258608 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c600be590 con 0x55868d774aa0 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 -- 172.21.15.53:0/1389258608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 msgr2=0x55868d766fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d766fd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0c600be930 tx=0x7f0c600d69e0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 -- 172.21.15.53:0/1389258608 shutdown_connections 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d77a600 0x55868d76c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d775490 0x55868d786400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/1389258608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d774aa0 0x55868d766fd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 -- 172.21.15.53:0/1389258608 >> 172.21.15.53:0/1389258608 conn(0x55868d76bd00 msgr2=0x55868d779ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 -- 172.21.15.53:0/1389258608 shutdown_connections 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 -- 172.21.15.53:0/1389258608 wait complete. 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 -- start start 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 Processor -- start 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d774aa0 0x55868d7843f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.477+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d7ed350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d77a600 0x55868d7ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55868d78c6b0 con 0x55868d77a600 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55868d78bfe0 con 0x55868d775490 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55868d78cd80 con 0x55868d774aa0 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 10 client.0 _ll_register_callbacks cb 0x7f0c64009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d77a600 0x55868d7ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d77a600 0x55868d7ef8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41500/0 (socket says 172.21.15.53:41500) 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 -- 172.21.15.53:0/664894159 learned_addr learned my addr 172.21.15.53:0/664894159 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c69d9b640 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d7ed350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d774aa0 0x55868d7843f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 -- 172.21.15.53:0/664894159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d774aa0 msgr2=0x55868d7843f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d774aa0 0x55868d7843f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 -- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 msgr2=0x55868d7ed350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d7ed350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 -- 172.21.15.53:0/664894159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55868d7f00d0 con 0x55868d77a600 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c6ad9d640 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d77a600 0x55868d7ef8a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009cd0 tx=0x7f0c64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bd451d0b148fe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c51ffb640 1 -- 172.21.15.53:0/664894159 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6402c060 con 0x55868d77a600 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c51ffb640 1 -- 172.21.15.53:0/664894159 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f0c6400f4e0 con 0x55868d77a600 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c6009cb40 con 0x55868d77a600 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c51ffb640 1 -- 172.21.15.53:0/664894159 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0c64004810 con 0x55868d77a600 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 10 client.15240 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.478+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55868d7f1410 con 0x55868d77a600 2024-04-28T12:50:35.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c51ffb640 1 -- 172.21.15.53:0/664894159 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c6009cb40 con 0x55868d77a600 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c51ffb640 1 -- 172.21.15.53:0/664894159 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0c64031070 con 0x55868d77a600 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c73dc4740 10 client.15240 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c73dc4740 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c73dc4740 20 client.15240 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d7f6020 0x55868d7f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c73dc4740 4 client.15240 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 --> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] -- command(tid 0: {"prefix": "status"}) -- 0x55868d6dd1c0 con 0x55868d7f6020 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.479+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d7f6020 0x55868d7f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c51ffb640 10 client.15240 ms_handle_connect on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c6a59c640 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d7f6020 0x55868d7f4d30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1edc7026391c7ff0 server_cookie=94da097899e1ca8f in_seq=0 out_seq=0 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c51ffb640 1 -- 172.21.15.53:0/664894159 <== mds.1 v2:172.21.15.77:6836/2147382100 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55868d6dd1c0 con 0x55868d7f6020 2024-04-28T12:50:35.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c51ffb640 10 client.15240 handle_command_reply: tid=0 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 client.15240 shutdown 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d77a600 msgr2=0x55868d7ef8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d77a600 0x55868d7ef8a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009cd0 tx=0x7f0c64004ab0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 shutdown_connections 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:6836/2147382100,v1:172.21.15.77:6837/2147382100] conn(0x55868d7f6020 0x55868d7f4d30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55868d77a600 0x55868d7ef8a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55868d775490 0x55868d7ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 --2- 172.21.15.53:0/664894159 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868d774aa0 0x55868d7843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c51ffb640 0 client.15240 ms_handle_reset on v2:172.21.15.77:6836/2147382100 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 >> 172.21.15.53:0/664894159 conn(0x55868d76bd00 msgr2=0x55868d76c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 shutdown_connections 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 1 -- 172.21.15.53:0/664894159 wait complete. 2024-04-28T12:50:35.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.480+0000 7f0c73dc4740 20 client.15240 trim_cache size 0 max 16384 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 1, 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24485, 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 107.947010283, 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 22, 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 88, 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 86, 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 128.36481877200001 2024-04-28T12:50:35.484 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0c540a3c40 msgr2=0x7f0c540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0c540a3c40 0x7f0c540a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55868d775cf0 tx=0x7f0c64009500 comp rx=0 tx=0).stop 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c6c1033f0 msgr2=0x7f0c6c198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c6c1033f0 0x7f0c6c198c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e9f0 tx=0x7f0c6000eeb0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 shutdown_connections 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f0c540a3c40 0x7f0c540a6100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6c108650 0x7f0c6c1a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c105d20 0x7f0c6c199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 --2- 172.21.15.53:0/2246998624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c6c1033f0 0x7f0c6c198c50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 >> 172.21.15.53:0/2246998624 conn(0x7f0c6c0fcde0 msgr2=0x7f0c6c1052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 shutdown_connections 2024-04-28T12:50:35.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.481+0000 7f0c70fc6640 1 -- 172.21.15.53:0/2246998624 wait complete. 2024-04-28T12:50:35.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-28T12:50:35.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 -- 172.21.15.53:0/614559221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac105d20 msgr2=0x7f4aac108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/614559221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac105d20 0x7f4aac108110 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009900 tx=0x7f4aa002a410 comp rx=0 tx=0).stop 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 -- 172.21.15.53:0/614559221 shutdown_connections 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/614559221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aac108650 0x7f4aac10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/614559221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac105d20 0x7f4aac108110 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/614559221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4aac1033f0 0x7f4aac1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 -- 172.21.15.53:0/614559221 >> 172.21.15.53:0/614559221 conn(0x7f4aac0fcde0 msgr2=0x7f4aac0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 -- 172.21.15.53:0/614559221 shutdown_connections 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 -- 172.21.15.53:0/614559221 wait complete. 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 Processor -- start 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.596+0000 7f4ab20f1640 1 -- start start 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac1033f0 0x7f4aac198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4aac105d20 0x7f4aac199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aac108650 0x7f4aac1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4aac10e490 con 0x7f4aac1033f0 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4aac10e310 con 0x7f4aac105d20 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4aac10e610 con 0x7f4aac108650 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac1033f0 0x7f4aac198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac1033f0 0x7f4aac198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41512/0 (socket says 172.21.15.53:41512) 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 -- 172.21.15.53:0/1364898538 learned_addr learned my addr 172.21.15.53:0/1364898538 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4aac105d20 0x7f4aac199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 -- 172.21.15.53:0/1364898538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aac108650 msgr2=0x7f4aac1a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aac108650 0x7f4aac1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 -- 172.21.15.53:0/1364898538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4aac105d20 msgr2=0x7f4aac199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4aac105d20 0x7f4aac199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 -- 172.21.15.53:0/1364898538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aac1a0840 con 0x7f4aac1033f0 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4aac105d20 0x7f4aac199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:35.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac1033f0 0x7f4aac198c30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00d950 tx=0x7f4a9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aa8ff9640 1 -- 172.21.15.53:0/1364898538 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c017070 con 0x7f4aac1033f0 2024-04-28T12:50:35.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aa8ff9640 1 -- 172.21.15.53:0/1364898538 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a9c009c30 con 0x7f4aac1033f0 2024-04-28T12:50:35.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aac1a0b30 con 0x7f4aac1033f0 2024-04-28T12:50:35.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4aa8ff9640 1 -- 172.21.15.53:0/1364898538 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c00f620 con 0x7f4aac1033f0 2024-04-28T12:50:35.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.597+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aac1a1080 con 0x7f4aac1033f0 2024-04-28T12:50:35.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.599+0000 7f4aa8ff9640 1 -- 172.21.15.53:0/1364898538 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4a9c010460 con 0x7f4aac1033f0 2024-04-28T12:50:35.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 Processor -- start 2024-04-28T12:50:35.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 -- start start 2024-04-28T12:50:35.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1605aa0 0x5627f1605ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1606490 0x5627f160af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f160da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5627f1566740 con 0x5627f1605aa0 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5627f15023a0 con 0x5627f1606490 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5627f1562700 con 0x5627f160b600 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1605aa0 0x5627f1605ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aa8ff9640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4a940a3c40 0x7f4a940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1605aa0 0x5627f1605ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41516/0 (socket says 172.21.15.53:41516) 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 -- 172.21.15.53:0/1278886787 learned_addr learned my addr 172.21.15.53:0/1278886787 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f160da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1606490 0x5627f160af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4a940a3c40 0x7f4a940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aa8ff9640 1 -- 172.21.15.53:0/1364898538 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f4a9c0d0340 con 0x7f4aac1033f0 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 -- 172.21.15.53:0/1278886787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 msgr2=0x5627f160da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f160da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 -- 172.21.15.53:0/1278886787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1606490 msgr2=0x5627f160af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1606490 0x5627f160af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.600+0000 7f4aabfff640 1 -- 172.21.15.53:0/1278886787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627f151e750 con 0x5627f1605aa0 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.601+0000 7f4aabfff640 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1605aa0 0x5627f1605ec0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4013850 tx=0x7f4aa4013d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dd8c2b8ec83603 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.601+0000 7f4a91ffb640 1 -- 172.21.15.53:0/1278886787 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4020070 con 0x5627f1605aa0 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.601+0000 7f4a91ffb640 1 -- 172.21.15.53:0/1278886787 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa401bc90 con 0x5627f1605aa0 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.601+0000 7f4a91ffb640 1 -- 172.21.15.53:0/1278886787 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40054f0 con 0x5627f1605aa0 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.601+0000 7f4ab4e06740 1 -- 172.21.15.53:0/1278886787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1605aa0 msgr2=0x5627f1605ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.601+0000 7f4aabfff640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4a940a3c40 0x7f4a940a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4aa401d000 tx=0x7f4aa401b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.601+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1605aa0 0x5627f1605ec0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4013850 tx=0x7f4aa4013d10 comp rx=0 tx=0).stop 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- 172.21.15.53:0/1278886787 shutdown_connections 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f160da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1606490 0x5627f160af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/1278886787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1605aa0 0x5627f1605ec0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- 172.21.15.53:0/1278886787 >> 172.21.15.53:0/1278886787 conn(0x5627f15fcd00 msgr2=0x5627f15fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- 172.21.15.53:0/1278886787 shutdown_connections 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- 172.21.15.53:0/1278886787 wait complete. 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- start start 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 Processor -- start 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f160fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f16173e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5627f160e5b0 con 0x5627f1606490 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5627f14edfe0 con 0x5627f160b600 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5627f160e730 con 0x5627f1605aa0 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4ab4e06740 10 client.0 _ll_register_callbacks cb 0x5627f1504150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4aaaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.602+0000 7f4aaaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41522/0 (socket says 172.21.15.53:41522) 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aaaffd640 1 -- 172.21.15.53:0/4169678664 learned_addr learned my addr 172.21.15.53:0/4169678664 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f160fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aabfff640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f16173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aaaffd640 1 -- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 msgr2=0x5627f160fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f160fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aaaffd640 1 -- 172.21.15.53:0/4169678664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 msgr2=0x5627f16173e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f16173e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aaaffd640 1 -- 172.21.15.53:0/4169678664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627f12782d0 con 0x5627f1606490 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aabfff640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f16173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f160fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16103a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4aac06a620 tx=0x7f4aa0032f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd11a238ad216f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4a8affd640 1 -- 172.21.15.53:0/4169678664 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0040070 con 0x5627f1606490 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4a8affd640 1 -- 172.21.15.53:0/4169678664 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f4aa0004240 con 0x5627f1606490 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aa401bc90 con 0x5627f1606490 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4a8affd640 1 -- 172.21.15.53:0/4169678664 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4aa0035040 con 0x5627f1606490 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4ab4e06740 10 client.15252 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5627f13426b0 con 0x5627f1606490 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.603+0000 7f4a8affd640 1 -- 172.21.15.53:0/4169678664 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa401bc90 con 0x5627f1606490 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4a8affd640 1 -- 172.21.15.53:0/4169678664 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4aa0004a20 con 0x5627f1606490 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4ab4e06740 10 client.15252 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4ab4e06740 10 client.15252 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4ab4e06740 20 client.15252 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f161c8f0 0x5627f161dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4ab4e06740 4 client.15252 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 --> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627f1574350 con 0x5627f161c8f0 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f161c8f0 0x5627f161dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4a8affd640 10 client.15252 ms_handle_connect on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.604+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f161c8f0 0x5627f161dd60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9056993f0975e8cf server_cookie=7ebe5e0e8d2aa797 in_seq=0 out_seq=0 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4a8affd640 1 -- 172.21.15.53:0/4169678664 <== mds.0 v2:172.21.15.191:6838/2399731075 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627f1574350 con 0x5627f161c8f0 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4a8affd640 10 client.15252 handle_command_reply: tid=0 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 client.15252 shutdown 2024-04-28T12:50:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 msgr2=0x5627f16103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16103a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4aac06a620 tx=0x7f4aa0032f00 comp rx=0 tx=0).stop 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 shutdown_connections 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f161c8f0 0x5627f161dd60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f16173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16103a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/4169678664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f160fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 >> 172.21.15.53:0/4169678664 conn(0x5627f15fcd00 msgr2=0x5627f15fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4a8affd640 0 client.15252 ms_handle_reset on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.605+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 shutdown_connections 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.606+0000 7f4ab4e06740 1 -- 172.21.15.53:0/4169678664 wait complete. 2024-04-28T12:50:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.606+0000 7f4ab4e06740 20 client.15252 trim_cache size 0 max 16384 2024-04-28T12:50:35.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 Processor -- start 2024-04-28T12:50:35.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 -- start start 2024-04-28T12:50:35.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1605aa0 0x5627f1606d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16173a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f1607820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5627f161c0a0 con 0x5627f1606490 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5627f161c770 con 0x5627f1605aa0 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5627f161ce40 con 0x5627f160b600 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16173a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41524/0 (socket says 172.21.15.53:41524) 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 -- 172.21.15.53:0/234621870 learned_addr learned my addr 172.21.15.53:0/234621870 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1605aa0 0x5627f1606d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aabfff640 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f1607820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 -- 172.21.15.53:0/234621870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 msgr2=0x5627f1607820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f1607820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 -- 172.21.15.53:0/234621870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1605aa0 msgr2=0x5627f1606d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1605aa0 0x5627f1606d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aaaffd640 1 -- 172.21.15.53:0/234621870 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627f1606870 con 0x5627f1606490 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.624+0000 7f4aabfff640 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f1607820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16173a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0006010 tx=0x7f4aa0003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4079778acb49b644 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4a917fa640 1 -- 172.21.15.53:0/234621870 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0040040 con 0x5627f1606490 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4a917fa640 1 -- 172.21.15.53:0/234621870 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa0004f90 con 0x5627f1606490 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4a917fa640 1 -- 172.21.15.53:0/234621870 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa004f6c0 con 0x5627f1606490 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 -- 172.21.15.53:0/234621870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 msgr2=0x5627f16173a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16173a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0006010 tx=0x7f4aa0003850 comp rx=0 tx=0).stop 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 -- 172.21.15.53:0/234621870 shutdown_connections 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f160b600 0x5627f1607820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f16173a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/234621870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f1605aa0 0x5627f1606d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 -- 172.21.15.53:0/234621870 >> 172.21.15.53:0/234621870 conn(0x5627f15fcd00 msgr2=0x5627f15fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 -- 172.21.15.53:0/234621870 shutdown_connections 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.625+0000 7f4ab4e06740 1 -- 172.21.15.53:0/234621870 wait complete. 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 -- start start 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 Processor -- start 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f16130d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f1613610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f1616630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5627f161dbe0 con 0x5627f1606490 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5627f161d510 con 0x5627f160b600 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5627f161e2b0 con 0x5627f1605aa0 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4ab4e06740 10 client.0 _ll_register_callbacks cb 0x7f4aa0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f1613610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f1613610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41532/0 (socket says 172.21.15.53:41532) 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 -- 172.21.15.53:0/2403674120 learned_addr learned my addr 172.21.15.53:0/2403674120 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f16130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aabfff640 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f1616630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 -- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 msgr2=0x5627f16130d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f16130d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 -- 172.21.15.53:0/2403674120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 msgr2=0x5627f1616630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f1616630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.626+0000 7f4aaaffd640 1 -- 172.21.15.53:0/2403674120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627f1616c20 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4aaaffd640 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f1613610 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005d00 tx=0x7f4aa0032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2cb232241bc910 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4a8affd640 1 -- 172.21.15.53:0/2403674120 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0040070 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4a8affd640 1 -- 172.21.15.53:0/2403674120 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f4aa0004ad0 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aa0004f90 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4a8affd640 1 -- 172.21.15.53:0/2403674120 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4aa0003670 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 10 client.15258 fetch_fsmap learned FSMap version 22 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5627f1682e00 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4a8affd640 1 -- 172.21.15.53:0/2403674120 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa0004f90 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4a8affd640 1 -- 172.21.15.53:0/2403674120 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4aa0056040 con 0x5627f1606490 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 10 client.15258 fetch_fsmap finished waiting for FSMap version 22 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 10 client.15258 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 20 client.15258 populate_metadata read hostname 'smithi053' 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f1687870 0x5627f1686580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 4 client.15258 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 --> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] -- command(tid 0: {"prefix": "status"}) -- 0x5627f156e1c0 con 0x5627f1687870 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.627+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f1687870 0x5627f1686580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4aab7fe640 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f1687870 0x5627f1686580 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73fa2906094facf1 server_cookie=dcca23eac2d315e5 in_seq=0 out_seq=0 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4a8affd640 10 client.15258 ms_handle_connect on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4a8affd640 1 -- 172.21.15.53:0/2403674120 <== mds.0 v2:172.21.15.191:6838/2399731075 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5627f156e1c0 con 0x5627f1687870 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4a8affd640 10 client.15258 handle_command_reply: tid=0 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 client.15258 shutdown 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 msgr2=0x5627f1613610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f1613610 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005d00 tx=0x7f4aa0032b20 comp rx=0 tx=0).stop 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 shutdown_connections 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:6838/2399731075,v1:172.21.15.191:6839/2399731075] conn(0x5627f1687870 0x5627f1686580 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5627f160b600 0x5627f1616630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4a8affd640 0 client.15258 ms_handle_reset on v2:172.21.15.191:6838/2399731075 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5627f1606490 0x5627f1613610 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 --2- 172.21.15.53:0/2403674120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5627f1605aa0 0x5627f16130d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 >> 172.21.15.53:0/2403674120 conn(0x5627f15fcd00 msgr2=0x5627f1606d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 shutdown_connections 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 1 -- 172.21.15.53:0/2403674120 wait complete. 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.628+0000 7f4ab4e06740 20 client.15258 trim_cache size 0 max 16384 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "10525726-055c-11ef-bc93-c7b262605968", 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24505, 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 111.114268025, 2024-04-28T12:50:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 22, 2024-04-28T12:50:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 88, 2024-04-28T12:50:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 86, 2024-04-28T12:50:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 125.607602394 2024-04-28T12:50:35.633 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-28T12:50:35.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4a940a3c40 msgr2=0x7f4a940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4a940a3c40 0x7f4a940a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4aa401d000 tx=0x7f4aa401b000 comp rx=0 tx=0).stop 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac1033f0 msgr2=0x7f4aac198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac1033f0 0x7f4aac198c30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00d950 tx=0x7f4a9c00de10 comp rx=0 tx=0).stop 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 shutdown_connections 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4a940a3c40 0x7f4a940a6100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aac108650 0x7f4aac1a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4aac105d20 0x7f4aac199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 --2- 172.21.15.53:0/1364898538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac1033f0 0x7f4aac198c30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 >> 172.21.15.53:0/1364898538 conn(0x7f4aac0fcde0 msgr2=0x7f4aac103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 shutdown_connections 2024-04-28T12:50:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:35.629+0000 7f4ab20f1640 1 -- 172.21.15.53:0/1364898538 wait complete. 2024-04-28T12:50:35.636 INFO:teuthology.run_tasks:Running task sequential... 2024-04-28T12:50:35.647 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-28T12:50:35.647 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-28T12:50:35.647 INFO:teuthology.task.print:Enabling mgr modules 2024-04-28T12:50:35.647 INFO:teuthology.run_tasks:Running task kclient... 2024-04-28T12:50:35.671 INFO:tasks.kclient:Mounting kernel clients... 2024-04-28T12:50:35.672 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-28T12:50:35.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:35.672 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-28T12:50:35.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:35.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-04-28T12:50:35.755 INFO:teuthology.orchestra.run.smithi053.stderr:Cannot find device "ceph-brx" 2024-04-28T12:50:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:50:35.756 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-28T12:50:35.756 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-28T12:50:35.756 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-28T12:50:35.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 10525726-055c-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-28T12:50:36.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:35 smithi191 ceph-mon[33016]: pgmap v348: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:35 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3914082184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:36.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:35 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4162936264' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:36.042 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:36.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:35 smithi053 ceph-mon[29182]: pgmap v348: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:36.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:35 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3914082184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:36.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:35 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4162936264' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:35 smithi077 ceph-mon[32553]: pgmap v348: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:35 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3914082184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:36.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:35 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4162936264' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:37.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 -- 172.21.15.53:0/1589474853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 msgr2=0x7f4df4102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:37.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 --2- 172.21.15.53:0/1589474853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df4102af0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4de80099c0 tx=0x7f4de8009cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:37.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 -- 172.21.15.53:0/1589474853 shutdown_connections 2024-04-28T12:50:37.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 --2- 172.21.15.53:0/1589474853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4df4103030 0x7f4df410f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:37.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 --2- 172.21.15.53:0/1589474853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4df41073a0 0x7f4df4107780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:37.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 --2- 172.21.15.53:0/1589474853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df4102af0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:37.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 -- 172.21.15.53:0/1589474853 >> 172.21.15.53:0/1589474853 conn(0x7f4df4078190 msgr2=0x7f4df40785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:37.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 -- 172.21.15.53:0/1589474853 shutdown_connections 2024-04-28T12:50:37.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.813+0000 7f4df899e700 1 -- 172.21.15.53:0/1589474853 wait complete. 2024-04-28T12:50:37.811 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:37 smithi053 ceph-mon[29182]: pgmap v349: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:37.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 Processor -- start 2024-04-28T12:50:37.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 -- start start 2024-04-28T12:50:37.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:37.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4df4103030 0x7f4df419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:37.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4df41073a0 0x7f4df4199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:37.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4df4112590 con 0x7f4df4102670 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4df4112410 con 0x7f4df4103030 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df899e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4df4112710 con 0x7f4df41073a0 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39384/0 (socket says 172.21.15.53:39384) 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.814+0000 7f4df259c700 1 -- 172.21.15.53:0/651523561 learned_addr learned my addr 172.21.15.53:0/651523561 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df1d9b700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4df4103030 0x7f4df419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df2d9d700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4df41073a0 0x7f4df4199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df259c700 1 -- 172.21.15.53:0/651523561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4df41073a0 msgr2=0x7f4df4199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df259c700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4df41073a0 0x7f4df4199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df259c700 1 -- 172.21.15.53:0/651523561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4df4103030 msgr2=0x7f4df419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df259c700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4df4103030 0x7f4df419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:37.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df259c700 1 -- 172.21.15.53:0/651523561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df419a070 con 0x7f4df4102670 2024-04-28T12:50:37.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df259c700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df419f750 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4de400e9a0 tx=0x7f4de400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:37.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4de37fe700 1 -- 172.21.15.53:0/651523561 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de400cbd0 con 0x7f4df4102670 2024-04-28T12:50:37.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4de37fe700 1 -- 172.21.15.53:0/651523561 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4de400cd70 con 0x7f4df4102670 2024-04-28T12:50:37.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4de37fe700 1 -- 172.21.15.53:0/651523561 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4018680 con 0x7f4df4102670 2024-04-28T12:50:37.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df419a360 con 0x7f4df4102670 2024-04-28T12:50:37.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.815+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df41a6620 con 0x7f4df4102670 2024-04-28T12:50:37.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.817+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df4103a90 con 0x7f4df4102670 2024-04-28T12:50:37.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.818+0000 7f4de37fe700 1 -- 172.21.15.53:0/651523561 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4de4010430 con 0x7f4df4102670 2024-04-28T12:50:37.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.818+0000 7f4de37fe700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ddc0a3cd0 0x7f4ddc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:37.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.818+0000 7f4df1d9b700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ddc0a3cd0 0x7f4ddc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:37.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.818+0000 7f4de37fe700 1 -- 172.21.15.53:0/651523561 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f4de40d07b0 con 0x7f4df4102670 2024-04-28T12:50:37.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.819+0000 7f4df1d9b700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ddc0a3cd0 0x7f4ddc0a6190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4df4078840 tx=0x7f4de8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:37.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:37.823+0000 7f4de37fe700 1 -- 172.21.15.53:0/651523561 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4de409d030 con 0x7f4df4102670 2024-04-28T12:50:38.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:37 smithi191 ceph-mon[33016]: pgmap v349: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:38.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.079+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4df402e490 con 0x7f4df4102670 2024-04-28T12:50:38.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.079+0000 7f4de37fe700 1 -- 172.21.15.53:0/651523561 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) ==== 53+0+73 (secure 0 0 0) 0x7f4de409c930 con 0x7f4df4102670 2024-04-28T12:50:38.076 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-28T12:50:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ddc0a3cd0 msgr2=0x7f4ddc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ddc0a3cd0 0x7f4ddc0a6190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4df4078840 tx=0x7f4de8005fb0 comp rx=0 tx=0).stop 2024-04-28T12:50:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 msgr2=0x7f4df419f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df419f750 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4de400e9a0 tx=0x7f4de400ecb0 comp rx=0 tx=0).stop 2024-04-28T12:50:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 shutdown_connections 2024-04-28T12:50:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f4ddc0a3cd0 0x7f4ddc0a6190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4df4102670 0x7f4df419f750 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4df41073a0 0x7f4df4199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 --2- 172.21.15.53:0/651523561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4df4103030 0x7f4df419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.081+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 >> 172.21.15.53:0/651523561 conn(0x7f4df4078190 msgr2=0x7f4df406be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.082+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 shutdown_connections 2024-04-28T12:50:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38.082+0000 7f4df899e700 1 -- 172.21.15.53:0/651523561 wait complete. 2024-04-28T12:50:38.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:37 smithi077 ceph-mon[32553]: pgmap v349: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-28T12:50:38.859 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-28T12:50:38.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:38.860 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-04-28T12:50:38.877 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-04-28T12:50:38.878 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 41907sec preferred_lft 41907sec 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:50:38.879 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-04-28T12:50:38.879 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-28T12:50:38.880 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:50:38.880 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-28T12:50:38.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add name ceph-brx type bridge 2024-04-28T12:50:38.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr flush dev ceph-brx 2024-04-28T12:50:38.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set ceph-brx up 2024-04-28T12:50:38.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-28T12:50:38.880 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-28T12:50:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:50:39.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:38 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/651523561' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:50:39.080 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:38 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/651523561' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:50:39.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:39.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:39.081 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-28T12:50:39.106 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-04-28T12:50:39.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:39.108 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-04-28T12:50:39.177 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-04-28T12:50:39.177 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-28T12:50:39.177 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-28T12:50:39.177 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-28T12:50:39.177 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-28T12:50:39.178 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:50:39.178 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-28T12:50:39.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-28T12:50:39.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-28T12:50:39.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-28T12:50:39.178 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-28T12:50:39.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:38 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/651523561' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:50:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:50:39.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:39.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:39.455 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-28T12:50:39.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:39.471 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2024-04-28T12:50:39.526 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:50:39.526 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-28T12:50:39.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-28T12:50:39.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-28T12:50:39.527 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-28T12:50:39.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:50:39.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:39.645 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-28T12:50:39.645 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:50:39.645 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-28T12:50:39.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-28T12:50:39.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-28T12:50:39.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-28T12:50:39.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-28T12:50:39.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-28T12:50:39.646 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-28T12:50:39.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:50:39.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:39.767 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:50:39.768 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-28T12:50:39.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-04-28T12:50:39.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.0 master ceph-brx 2024-04-28T12:50:39.768 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-28T12:50:39.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:50:39.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:39.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T12:50:39.846 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-28T12:50:39.861 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-28T12:50:39.861 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T12:50:39.861 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-28T12:50:39.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:39 smithi053 ceph-mon[29182]: pgmap v350: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:39.910 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-28T12:50:39.910 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-28T12:50:39.910 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/068a4084-e7f8-4632-9d3e-602add1a0061 2024-04-28T12:50:39.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:39.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/068a4084-e7f8-4632-9d3e-602add1a0061 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-28T12:50:40.004 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-28T12:50:40.004 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-28T12:50:40.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:39 smithi191 ceph-mon[33016]: pgmap v350: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-28T12:50:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-28T12:50:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: resolved to: "172.21.15.53:6789,172.21.15.77:6789,172.21.15.191:6789" 2024-04-28T12:50:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: trying mount with new device syntax: 0@10525726-055c-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/068a4084-e7f8-4632-9d3e-602add1a0061 2024-04-28T12:50:40.103 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.53:6789/172.21.15.77:6789/172.21.15.191:6789" will pass to kernel 2024-04-28T12:50:40.104 INFO:teuthology.orchestra.run.smithi053.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-28T12:50:40.104 INFO:teuthology.orchestra.run.smithi053.stdout: You just mounted a file system that supports labels which does not 2024-04-28T12:50:40.104 INFO:teuthology.orchestra.run.smithi053.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-28T12:50:40.104 INFO:teuthology.orchestra.run.smithi053.stdout: applications will generate AVC messages and not be allowed access to 2024-04-28T12:50:40.104 INFO:teuthology.orchestra.run.smithi053.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-28T12:50:40.105 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-28T12:50:40.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:50:40.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-28T12:50:40.131 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T12:50:40.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:39 smithi077 ceph-mon[32553]: pgmap v350: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:40.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T12:50:40.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:40.229 INFO:teuthology.orchestra.run.smithi053.stdout:client24986 2024-04-28T12:50:40.230 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T12:50:40.230 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T12:50:40.247 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T12:50:40.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T12:50:40.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:40.344 INFO:teuthology.orchestra.run.smithi053.stdout:client24986 2024-04-28T12:50:40.345 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T12:50:40.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T12:50:40.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:client24986 2024-04-28T12:50:40.402 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/10525726-055c-11ef-bc93-c7b262605968.client24986/status') 2024-04-28T12:50:40.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:50:40.460 INFO:teuthology.orchestra.run.smithi053.stdout:instance: client.24986 (3)172.21.15.53:0/3413655229 2024-04-28T12:50:40.460 INFO:teuthology.orchestra.run.smithi053.stdout:blocklisted: false 2024-04-28T12:50:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records in 2024-04-28T12:50:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records out 2024-04-28T12:50:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:71 bytes copied, 2.4415e-05 s, 2.9 MB/s 2024-04-28T12:50:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:50:40.468 INFO:teuthology.run_tasks:Running task exec... 2024-04-28T12:50:40.479 INFO:teuthology.task.exec:Executing custom commands... 2024-04-28T12:50:40.479 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-28T12:50:40.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 -- 172.21.15.53:0/1833357509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458102a80 msgr2=0x7f4458102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 --2- 172.21.15.53:0/1833357509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458102a80 0x7f4458102f00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f444c009900 tx=0x7f444c02a410 comp rx=0 tx=0).stop 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 -- 172.21.15.53:0/1833357509 shutdown_connections 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 --2- 172.21.15.53:0/1833357509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4458103440 0x7f4458109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 --2- 172.21.15.53:0/1833357509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458102a80 0x7f4458102f00 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 --2- 172.21.15.53:0/1833357509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4458101890 0x7f4458101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 -- 172.21.15.53:0/1833357509 >> 172.21.15.53:0/1833357509 conn(0x7f44580fce00 msgr2=0x7f44580ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 -- 172.21.15.53:0/1833357509 shutdown_connections 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 -- 172.21.15.53:0/1833357509 wait complete. 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.586+0000 7f445e1f7640 1 Processor -- start 2024-04-28T12:50:40.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f445e1f7640 1 -- start start 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f445e1f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4458101890 0x7f4458105820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f445e1f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4458102a80 0x7f4458103e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f445e1f7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458103440 0x7f44581043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f445e1f7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f445810d3c0 con 0x7f4458103440 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f445e1f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f445810d240 con 0x7f4458102a80 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f445e1f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f445810d540 con 0x7f4458101890 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458103440 0x7f44581043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458103440 0x7f44581043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39404/0 (socket says 172.21.15.53:39404) 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 -- 172.21.15.53:0/2190030812 learned_addr learned my addr 172.21.15.53:0/2190030812 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f44577fe640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4458101890 0x7f4458105820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4456ffd640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4458102a80 0x7f4458103e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 -- 172.21.15.53:0/2190030812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4458101890 msgr2=0x7f4458105820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4458101890 0x7f4458105820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 -- 172.21.15.53:0/2190030812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4458102a80 msgr2=0x7f4458103e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4458102a80 0x7f4458103e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 -- 172.21.15.53:0/2190030812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4458104c30 con 0x7f4458103440 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.587+0000 7f4457fff640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458103440 0x7f44581043a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f445000eb10 tx=0x7f445000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.588+0000 7f4454ff9640 1 -- 172.21.15.53:0/2190030812 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4450004b20 con 0x7f4458103440 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.588+0000 7f4454ff9640 1 -- 172.21.15.53:0/2190030812 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4450004cc0 con 0x7f4458103440 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.588+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44581a2c40 con 0x7f4458103440 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.588+0000 7f4454ff9640 1 -- 172.21.15.53:0/2190030812 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4450010670 con 0x7f4458103440 2024-04-28T12:50:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.588+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44581a3190 con 0x7f4458103440 2024-04-28T12:50:40.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.589+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44581a33c0 con 0x7f4458103440 2024-04-28T12:50:40.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.590+0000 7f4454ff9640 1 -- 172.21.15.53:0/2190030812 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4450005020 con 0x7f4458103440 2024-04-28T12:50:40.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.590+0000 7f4454ff9640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f44400a3cf0 0x7f44400a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.590+0000 7f4454ff9640 1 -- 172.21.15.53:0/2190030812 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f44500d0ca0 con 0x7f4458103440 2024-04-28T12:50:40.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.590+0000 7f44577fe640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f44400a3cf0 0x7f44400a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.591+0000 7f44577fe640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f44400a3cf0 0x7f44400a61b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4448005db0 tx=0x7f4448005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:40.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.595+0000 7f4454ff9640 1 -- 172.21.15.53:0/2190030812 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f445009d520 con 0x7f4458103440 2024-04-28T12:50:40.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:40.824+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f44581a3590 con 0x7f4458103440 2024-04-28T12:50:41.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.782+0000 7f4454ff9640 1 -- 172.21.15.53:0/2190030812 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v23) ==== 118+0+0 (secure 0 0 0) 0x7f445009ce20 con 0x7f4458103440 2024-04-28T12:50:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f44400a3cf0 msgr2=0x7f44400a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f44400a3cf0 0x7f44400a61b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4448005db0 tx=0x7f4448005d40 comp rx=0 tx=0).stop 2024-04-28T12:50:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458103440 msgr2=0x7f44581043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458103440 0x7f44581043a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f445000eb10 tx=0x7f445000c2d0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 shutdown_connections 2024-04-28T12:50:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f44400a3cf0 0x7f44400a61b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4458103440 0x7f44581043a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4458102a80 0x7f4458103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 --2- 172.21.15.53:0/2190030812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4458101890 0x7f4458105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.785+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 >> 172.21.15.53:0/2190030812 conn(0x7f44580fce00 msgr2=0x7f44580fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.786+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 shutdown_connections 2024-04-28T12:50:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.786+0000 7f445e1f7640 1 -- 172.21.15.53:0/2190030812 wait complete. 2024-04-28T12:50:41.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-28T12:50:41.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 -- 172.21.15.53:0/1085237751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0758101890 msgr2=0x7f0758101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:41.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 --2- 172.21.15.53:0/1085237751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0758101890 0x7f0758101cb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0754009900 tx=0x7f075402a410 comp rx=0 tx=0).stop 2024-04-28T12:50:41.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 -- 172.21.15.53:0/1085237751 shutdown_connections 2024-04-28T12:50:41.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 --2- 172.21.15.53:0/1085237751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0758103440 0x7f0758109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 --2- 172.21.15.53:0/1085237751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0758102a80 0x7f0758102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 --2- 172.21.15.53:0/1085237751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0758101890 0x7f0758101cb0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 -- 172.21.15.53:0/1085237751 >> 172.21.15.53:0/1085237751 conn(0x7f07580fce00 msgr2=0x7f07580ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 -- 172.21.15.53:0/1085237751 shutdown_connections 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.910+0000 7f075eecc640 1 -- 172.21.15.53:0/1085237751 wait complete. 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 Processor -- start 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 -- start start 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0758101890 0x7f075806ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0758102a80 0x7f075819d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0758103440 0x7f075819fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f075810d390 con 0x7f0758102a80 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f075810d210 con 0x7f0758103440 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075eecc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f075810d510 con 0x7f0758101890 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f074ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0758102a80 0x7f075819d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:41.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f074ffff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0758102a80 0x7f075819d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39422/0 (socket says 172.21.15.53:39422) 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f074ffff640 1 -- 172.21.15.53:0/1459642607 learned_addr learned my addr 172.21.15.53:0/1459642607 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075d469640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0758103440 0x7f075819fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f075cc68640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0758101890 0x7f075806ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f074ffff640 1 -- 172.21.15.53:0/1459642607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0758101890 msgr2=0x7f075806ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f074ffff640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0758101890 0x7f075806ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f074ffff640 1 -- 172.21.15.53:0/1459642607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0758103440 msgr2=0x7f075819fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.911+0000 7f074ffff640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0758103440 0x7f075819fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.912+0000 7f074ffff640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07581a0780 con 0x7f0758102a80 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.912+0000 7f074ffff640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0758102a80 0x7f075819d880 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f074800d950 tx=0x7f074800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.912+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748017070 con 0x7f0758102a80 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.912+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0748009c30 con 0x7f0758102a80 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.912+0000 7f075eecc640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07581a0a70 con 0x7f0758102a80 2024-04-28T12:50:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.912+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074800f620 con 0x7f0758102a80 2024-04-28T12:50:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.912+0000 7f075eecc640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07581a0fc0 con 0x7f0758102a80 2024-04-28T12:50:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.913+0000 7f075eecc640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0758108c80 con 0x7f0758102a80 2024-04-28T12:50:41.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.914+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0748010460 con 0x7f0758102a80 2024-04-28T12:50:41.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.915+0000 7f074dffb640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400a3cf0 0x7f07400a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:41.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.915+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f07480d01c0 con 0x7f0758102a80 2024-04-28T12:50:41.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.915+0000 7f075cc68640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400a3cf0 0x7f07400a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:41.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.915+0000 7f075cc68640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400a3cf0 0x7f07400a61b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0754000c00 tx=0x7f07540314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:41.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:41.920+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f074809ca40 con 0x7f0758102a80 2024-04-28T12:50:42.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:41 smithi191 ceph-mon[33016]: pgmap v351: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:42.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:41 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2190030812' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T12:50:42.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:50:42.133+0000 7f075eecc640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f0758062d70 con 0x7f07400a3cf0 2024-04-28T12:50:42.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:41 smithi053 ceph-mon[29182]: pgmap v351: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:42.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:41 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2190030812' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T12:50:42.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:41 smithi077 ceph-mon[32553]: pgmap v351: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:50:42.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:41 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2190030812' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T12:50:43.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:42 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2190030812' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T12:50:43.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:42 smithi191 ceph-mon[33016]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:50:43.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:42 smithi191 ceph-mon[33016]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:43.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:42 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2190030812' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T12:50:43.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:42 smithi053 ceph-mon[29182]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:50:43.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:42 smithi053 ceph-mon[29182]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:43.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:42 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2190030812' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T12:50:43.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:42 smithi077 ceph-mon[32553]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:50:43.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:42 smithi077 ceph-mon[32553]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:43 smithi191 ceph-mon[33016]: pgmap v352: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:44.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:43 smithi053 ceph-mon[29182]: pgmap v352: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:44.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:43 smithi077 ceph-mon[32553]: pgmap v352: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:50:46.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:45 smithi053 ceph-mon[29182]: pgmap v353: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:50:46.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:45 smithi077 ceph-mon[32553]: pgmap v353: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:50:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:45 smithi191 ceph-mon[33016]: pgmap v353: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:50:48.090 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:47 smithi191 ceph-mon[33016]: pgmap v354: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:50:48.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:47 smithi053 ceph-mon[29182]: pgmap v354: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:50:48.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:47 smithi077 ceph-mon[32553]: pgmap v354: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:50:50.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:49 smithi053 ceph-mon[29182]: pgmap v355: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-28T12:50:50.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:49 smithi077 ceph-mon[32553]: pgmap v355: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-28T12:50:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:49 smithi191 ceph-mon[33016]: pgmap v355: 97 pgs: 97 active+clean; 630 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-28T12:50:52.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:51 smithi053 ceph-mon[29182]: pgmap v356: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:50:52.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:51 smithi077 ceph-mon[32553]: pgmap v356: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:50:52.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:51 smithi191 ceph-mon[33016]: pgmap v356: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:50:54.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:53 smithi053 ceph-mon[29182]: pgmap v357: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:50:54.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:53 smithi077 ceph-mon[32553]: pgmap v357: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:50:54.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:53 smithi191 ceph-mon[33016]: pgmap v357: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:50:56.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:55 smithi053 ceph-mon[29182]: pgmap v358: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:50:56.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:55 smithi077 ceph-mon[32553]: pgmap v358: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:50:56.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:55 smithi191 ceph-mon[33016]: pgmap v358: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:50:58.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:57 smithi053 ceph-mon[29182]: pgmap v359: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-28T12:50:58.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:57 smithi077 ceph-mon[32553]: pgmap v359: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-28T12:50:58.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:57 smithi191 ceph-mon[33016]: pgmap v359: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-28T12:51:00.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:50:59 smithi053 ceph-mon[29182]: pgmap v360: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-28T12:51:00.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:50:59 smithi077 ceph-mon[32553]: pgmap v360: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-28T12:51:00.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:50:59 smithi191 ceph-mon[33016]: pgmap v360: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-28T12:51:02.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:01 smithi053 ceph-mon[29182]: pgmap v361: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-28T12:51:02.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:01 smithi077 ceph-mon[32553]: pgmap v361: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-28T12:51:02.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:01 smithi191 ceph-mon[33016]: pgmap v361: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-28T12:51:02.992 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:02 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:51:03.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:02 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:51:03.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:02 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:51:04.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:03 smithi053 ceph-mon[29182]: pgmap v362: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:03 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:03 smithi077 ceph-mon[32553]: pgmap v362: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:04.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:03 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:03 smithi191 ceph-mon[33016]: pgmap v362: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:04.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:04.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:03 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:05.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:51:05.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:51:05.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:51:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:51:05.198 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:05.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:51:05.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:51:05.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:51:06.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:06 smithi191 ceph-mon[33016]: pgmap v363: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:06.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:06 smithi053 ceph-mon[29182]: pgmap v363: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:06.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:06 smithi077 ceph-mon[32553]: pgmap v363: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:08 smithi191 ceph-mon[33016]: pgmap v364: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:08.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:08 smithi053 ceph-mon[29182]: pgmap v364: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:08.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:08 smithi077 ceph-mon[32553]: pgmap v364: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:10 smithi191 ceph-mon[33016]: pgmap v365: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:10.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:10 smithi053 ceph-mon[29182]: pgmap v365: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:10.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:10 smithi077 ceph-mon[32553]: pgmap v365: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:12.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:12 smithi191 ceph-mon[33016]: pgmap v366: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:12.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:12 smithi053 ceph-mon[29182]: pgmap v366: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:12.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:12 smithi077 ceph-mon[32553]: pgmap v366: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:14.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:14 smithi053 ceph-mon[29182]: pgmap v367: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:14.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:14 smithi077 ceph-mon[32553]: pgmap v367: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:14.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:14 smithi191 ceph-mon[33016]: pgmap v367: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:16.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:16 smithi053 ceph-mon[29182]: pgmap v368: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:16.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:16 smithi077 ceph-mon[32553]: pgmap v368: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:16.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:16 smithi191 ceph-mon[33016]: pgmap v368: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:18.590 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:18 smithi191 ceph-mon[33016]: pgmap v369: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:18.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:18 smithi053 ceph-mon[29182]: pgmap v369: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:18.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:18 smithi077 ceph-mon[32553]: pgmap v369: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:20.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:20 smithi053 ceph-mon[29182]: pgmap v370: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:20.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:20 smithi077 ceph-mon[32553]: pgmap v370: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:20 smithi191 ceph-mon[33016]: pgmap v370: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:22.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:22 smithi053 ceph-mon[29182]: pgmap v371: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:22.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:22 smithi077 ceph-mon[32553]: pgmap v371: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:22.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:22 smithi191 ceph-mon[33016]: pgmap v371: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:24.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:24 smithi053 ceph-mon[29182]: pgmap v372: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:24.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:24 smithi077 ceph-mon[32553]: pgmap v372: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:24.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:24 smithi191 ceph-mon[33016]: pgmap v372: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:26.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:26 smithi053 ceph-mon[29182]: pgmap v373: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:26.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:26 smithi077 ceph-mon[32553]: pgmap v373: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:26.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:26 smithi191 ceph-mon[33016]: pgmap v373: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:28.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:28 smithi053 ceph-mon[29182]: pgmap v374: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:28.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:28 smithi077 ceph-mon[32553]: pgmap v374: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:28.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:28 smithi191 ceph-mon[33016]: pgmap v374: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:30.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:30 smithi053 ceph-mon[29182]: pgmap v375: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:30.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:30 smithi077 ceph-mon[32553]: pgmap v375: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:30.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:30 smithi191 ceph-mon[33016]: pgmap v375: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:32.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:32 smithi053 ceph-mon[29182]: pgmap v376: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:32.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:32 smithi077 ceph-mon[32553]: pgmap v376: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:32.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:32 smithi191 ceph-mon[33016]: pgmap v376: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:34.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:34 smithi053 ceph-mon[29182]: pgmap v377: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:34.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:34 smithi077 ceph-mon[32553]: pgmap v377: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:34.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:34 smithi191 ceph-mon[33016]: pgmap v377: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:36.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:36 smithi053 ceph-mon[29182]: pgmap v378: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:36.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:36 smithi077 ceph-mon[32553]: pgmap v378: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:36.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:36 smithi191 ceph-mon[33016]: pgmap v378: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:38.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:38 smithi053 ceph-mon[29182]: pgmap v379: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:38.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:38 smithi077 ceph-mon[32553]: pgmap v379: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:38.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:38 smithi191 ceph-mon[33016]: pgmap v379: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:40.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:40 smithi053 ceph-mon[29182]: pgmap v380: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:40.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:40 smithi077 ceph-mon[32553]: pgmap v380: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:40.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:40 smithi191 ceph-mon[33016]: pgmap v380: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:42.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:42 smithi053 ceph-mon[29182]: pgmap v381: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:42.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:42 smithi077 ceph-mon[32553]: pgmap v381: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:42.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:42 smithi191 ceph-mon[33016]: pgmap v381: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:44.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:44 smithi053 ceph-mon[29182]: pgmap v382: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:44.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:44 smithi077 ceph-mon[32553]: pgmap v382: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:44 smithi191 ceph-mon[33016]: pgmap v382: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:46.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:46 smithi053 ceph-mon[29182]: pgmap v383: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:46.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:46 smithi077 ceph-mon[32553]: pgmap v383: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:46.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:46 smithi191 ceph-mon[33016]: pgmap v383: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:48.590 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:48 smithi191 ceph-mon[33016]: pgmap v384: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:48.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:48 smithi053 ceph-mon[29182]: pgmap v384: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:48.698 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:48 smithi077 ceph-mon[32553]: pgmap v384: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:50.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:50 smithi053 ceph-mon[29182]: pgmap v385: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:50.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:50 smithi077 ceph-mon[32553]: pgmap v385: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:50.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:50 smithi191 ceph-mon[33016]: pgmap v385: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:52.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:52 smithi053 ceph-mon[29182]: pgmap v386: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:52.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:52 smithi077 ceph-mon[32553]: pgmap v386: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:52.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:52 smithi191 ceph-mon[33016]: pgmap v386: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:53.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:53 smithi053 ceph-mon[29182]: pgmap v387: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:53.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:53 smithi077 ceph-mon[32553]: pgmap v387: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:53 smithi191 ceph-mon[33016]: pgmap v387: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:55.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:55 smithi053 ceph-mon[29182]: pgmap v388: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:55.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:55 smithi077 ceph-mon[32553]: pgmap v388: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:56.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:55 smithi191 ceph-mon[33016]: pgmap v388: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:57.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:57 smithi053 ceph-mon[29182]: pgmap v389: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:57.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:57 smithi077 ceph-mon[32553]: pgmap v389: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:58.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:57 smithi191 ceph-mon[33016]: pgmap v389: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:59.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:51:59 smithi053 ceph-mon[29182]: pgmap v390: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:51:59 smithi077 ceph-mon[32553]: pgmap v390: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:51:59 smithi191 ceph-mon[33016]: pgmap v390: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:01.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:01 smithi053 ceph-mon[29182]: pgmap v391: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:01.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:01 smithi077 ceph-mon[32553]: pgmap v391: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:02.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:01 smithi191 ceph-mon[33016]: pgmap v391: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:03.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:03 smithi053 ceph-mon[29182]: pgmap v392: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:03.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:03 smithi077 ceph-mon[32553]: pgmap v392: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:03 smithi191 ceph-mon[33016]: pgmap v392: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:04.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:04 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:52:04.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:04 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:52:05.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:04 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:52:06.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: pgmap v393: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:52:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:52:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: pgmap v393: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:52:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: pgmap v393: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:52:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:52:08.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:08 smithi191 ceph-mon[33016]: pgmap v394: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:08.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:08 smithi053 ceph-mon[29182]: pgmap v394: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:08.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:08 smithi077 ceph-mon[32553]: pgmap v394: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:10.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:10 smithi191 ceph-mon[33016]: pgmap v395: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:10.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:10 smithi053 ceph-mon[29182]: pgmap v395: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:10.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:10 smithi077 ceph-mon[32553]: pgmap v395: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:12.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:12 smithi191 ceph-mon[33016]: pgmap v396: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:12.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:12 smithi053 ceph-mon[29182]: pgmap v396: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:12.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:12 smithi077 ceph-mon[32553]: pgmap v396: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:14.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:14 smithi191 ceph-mon[33016]: pgmap v397: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:52:14.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:14 smithi053 ceph-mon[29182]: pgmap v397: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:52:14.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:14 smithi077 ceph-mon[32553]: pgmap v397: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:52:16.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:16 smithi191 ceph-mon[33016]: pgmap v398: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:16.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:16 smithi053 ceph-mon[29182]: pgmap v398: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:16.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:16 smithi077 ceph-mon[32553]: pgmap v398: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:18.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:18 smithi191 ceph-mon[33016]: pgmap v399: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:52:18.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:18 smithi053 ceph-mon[29182]: pgmap v399: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:52:18.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:18 smithi077 ceph-mon[32553]: pgmap v399: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:52:20.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:20 smithi191 ceph-mon[33016]: pgmap v400: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:52:20.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:20 smithi053 ceph-mon[29182]: pgmap v400: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:52:20.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:20 smithi077 ceph-mon[32553]: pgmap v400: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:52:22.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:22 smithi191 ceph-mon[33016]: pgmap v401: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:22.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:22 smithi053 ceph-mon[29182]: pgmap v401: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:22.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:22 smithi077 ceph-mon[32553]: pgmap v401: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:52:24.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:24 smithi191 ceph-mon[33016]: pgmap v402: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:24.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:24 smithi053 ceph-mon[29182]: pgmap v402: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:24.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:24 smithi077 ceph-mon[32553]: pgmap v402: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:26.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:26 smithi191 ceph-mon[33016]: pgmap v403: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:26.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:26 smithi053 ceph-mon[29182]: pgmap v403: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:26.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:26 smithi077 ceph-mon[32553]: pgmap v403: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:28.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:28 smithi191 ceph-mon[33016]: pgmap v404: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:28.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:28 smithi053 ceph-mon[29182]: pgmap v404: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:28.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:28 smithi077 ceph-mon[32553]: pgmap v404: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:30.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:30 smithi191 ceph-mon[33016]: pgmap v405: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:30.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:30 smithi053 ceph-mon[29182]: pgmap v405: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:30.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:30 smithi077 ceph-mon[32553]: pgmap v405: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:32.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:32 smithi053 ceph-mon[29182]: pgmap v406: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:32.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:32 smithi077 ceph-mon[32553]: pgmap v406: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:32.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:32 smithi191 ceph-mon[33016]: pgmap v406: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:34.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:34 smithi053 ceph-mon[29182]: pgmap v407: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:34.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:34 smithi077 ceph-mon[32553]: pgmap v407: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:34.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:34 smithi191 ceph-mon[33016]: pgmap v407: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:36.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:36 smithi053 ceph-mon[29182]: pgmap v408: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:36.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:36 smithi077 ceph-mon[32553]: pgmap v408: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:36.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:36 smithi191 ceph-mon[33016]: pgmap v408: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:38.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:38 smithi053 ceph-mon[29182]: pgmap v409: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:38.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:38 smithi077 ceph-mon[32553]: pgmap v409: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:38.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:38 smithi191 ceph-mon[33016]: pgmap v409: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:40.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:40 smithi053 ceph-mon[29182]: pgmap v410: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:40.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:40 smithi077 ceph-mon[32553]: pgmap v410: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:40.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:40 smithi191 ceph-mon[33016]: pgmap v410: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:42.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:42 smithi053 ceph-mon[29182]: pgmap v411: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:42.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:42 smithi077 ceph-mon[32553]: pgmap v411: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:42.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:42 smithi191 ceph-mon[33016]: pgmap v411: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:44.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:44 smithi053 ceph-mon[29182]: pgmap v412: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:44.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:44 smithi077 ceph-mon[32553]: pgmap v412: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:44.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:44 smithi191 ceph-mon[33016]: pgmap v412: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:46.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:46 smithi053 ceph-mon[29182]: pgmap v413: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:46.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:46 smithi077 ceph-mon[32553]: pgmap v413: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:46.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:46 smithi191 ceph-mon[33016]: pgmap v413: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:48.341 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:48 smithi191 ceph-mon[33016]: pgmap v414: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:48.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:48 smithi053 ceph-mon[29182]: pgmap v414: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:48.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:48 smithi077 ceph-mon[32553]: pgmap v414: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:50.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:50 smithi053 ceph-mon[29182]: pgmap v415: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:50.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:50 smithi077 ceph-mon[32553]: pgmap v415: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:50.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:50 smithi191 ceph-mon[33016]: pgmap v415: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:52.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:52 smithi053 ceph-mon[29182]: pgmap v416: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:52.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:52 smithi077 ceph-mon[32553]: pgmap v416: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:52.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:52 smithi191 ceph-mon[33016]: pgmap v416: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:52:54.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:54 smithi053 ceph-mon[29182]: pgmap v417: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:54.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:54 smithi077 ceph-mon[32553]: pgmap v417: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:54.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:54 smithi191 ceph-mon[33016]: pgmap v417: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:56.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:56 smithi053 ceph-mon[29182]: pgmap v418: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:56.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:56 smithi077 ceph-mon[32553]: pgmap v418: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:56.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:56 smithi191 ceph-mon[33016]: pgmap v418: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:58.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:52:58 smithi053 ceph-mon[29182]: pgmap v419: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:58.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:52:58 smithi077 ceph-mon[32553]: pgmap v419: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:58.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:52:58 smithi191 ceph-mon[33016]: pgmap v419: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:00.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:00 smithi053 ceph-mon[29182]: pgmap v420: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:00.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:00 smithi077 ceph-mon[32553]: pgmap v420: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:00.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:00 smithi191 ceph-mon[33016]: pgmap v420: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:02.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:02 smithi053 ceph-mon[29182]: pgmap v421: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:02.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:02 smithi077 ceph-mon[32553]: pgmap v421: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:02.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:02 smithi191 ceph-mon[33016]: pgmap v421: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:04.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:04 smithi053 ceph-mon[29182]: pgmap v422: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:04.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:04 smithi077 ceph-mon[32553]: pgmap v422: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:04.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:04 smithi191 ceph-mon[33016]: pgmap v422: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:06 smithi053 ceph-mon[29182]: pgmap v423: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:06.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:06 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:53:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:06 smithi077 ceph-mon[32553]: pgmap v423: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:06 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:53:06.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:06 smithi191 ceph-mon[33016]: pgmap v423: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:06.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:06 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:53:07.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:07 smithi053 ceph-mon[29182]: pgmap v424: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:07.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:53:07.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:53:07.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:07 smithi077 ceph-mon[32553]: pgmap v424: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:07.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:53:07.699 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:53:07.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:07.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:07 smithi191 ceph-mon[33016]: pgmap v424: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:07.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:53:07.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:53:07.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:53:09.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:09 smithi053 ceph-mon[29182]: pgmap v425: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:09.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:09 smithi077 ceph-mon[32553]: pgmap v425: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:10.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:09 smithi191 ceph-mon[33016]: pgmap v425: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:11.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:11 smithi053 ceph-mon[29182]: pgmap v426: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:11.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:11 smithi077 ceph-mon[32553]: pgmap v426: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:12.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:11 smithi191 ceph-mon[33016]: pgmap v426: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:13.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:13 smithi053 ceph-mon[29182]: pgmap v427: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:13.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:13 smithi077 ceph-mon[32553]: pgmap v427: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:14.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:13 smithi191 ceph-mon[33016]: pgmap v427: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:15.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:15 smithi053 ceph-mon[29182]: pgmap v428: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:15.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:15 smithi077 ceph-mon[32553]: pgmap v428: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:16.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:15 smithi191 ceph-mon[33016]: pgmap v428: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:17.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:17 smithi053 ceph-mon[29182]: pgmap v429: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:17.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:17 smithi077 ceph-mon[32553]: pgmap v429: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:18.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:17 smithi191 ceph-mon[33016]: pgmap v429: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:19.841 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:19 smithi191 ceph-mon[33016]: pgmap v430: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:19.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:19 smithi053 ceph-mon[29182]: pgmap v430: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:19.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:19 smithi077 ceph-mon[32553]: pgmap v430: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:21.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:21 smithi053 ceph-mon[29182]: pgmap v431: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:21.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:21 smithi077 ceph-mon[32553]: pgmap v431: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:22.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:21 smithi191 ceph-mon[33016]: pgmap v431: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:23.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:23 smithi053 ceph-mon[29182]: pgmap v432: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:23.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:23 smithi077 ceph-mon[32553]: pgmap v432: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:24.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:23 smithi191 ceph-mon[33016]: pgmap v432: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:25.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:25 smithi053 ceph-mon[29182]: pgmap v433: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:25.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:25 smithi077 ceph-mon[32553]: pgmap v433: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:26.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:25 smithi191 ceph-mon[33016]: pgmap v433: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:27.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:27 smithi053 ceph-mon[29182]: pgmap v434: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:27.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:27 smithi077 ceph-mon[32553]: pgmap v434: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:28.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:27 smithi191 ceph-mon[33016]: pgmap v434: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:29.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:29 smithi053 ceph-mon[29182]: pgmap v435: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:29.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:29 smithi077 ceph-mon[32553]: pgmap v435: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:30.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:29 smithi191 ceph-mon[33016]: pgmap v435: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:31.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:31 smithi053 ceph-mon[29182]: pgmap v436: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:31.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:31 smithi077 ceph-mon[32553]: pgmap v436: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:32.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:31 smithi191 ceph-mon[33016]: pgmap v436: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:33.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:33 smithi053 ceph-mon[29182]: pgmap v437: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:33.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:33 smithi077 ceph-mon[32553]: pgmap v437: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:34.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:33 smithi191 ceph-mon[33016]: pgmap v437: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:35.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:35 smithi053 ceph-mon[29182]: pgmap v438: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:35.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:35 smithi077 ceph-mon[32553]: pgmap v438: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:36.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:35 smithi191 ceph-mon[33016]: pgmap v438: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:37.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:37 smithi053 ceph-mon[29182]: pgmap v439: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:37.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:37 smithi077 ceph-mon[32553]: pgmap v439: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:38.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:37 smithi191 ceph-mon[33016]: pgmap v439: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:39.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:39 smithi053 ceph-mon[29182]: pgmap v440: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:39.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:39 smithi077 ceph-mon[32553]: pgmap v440: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:40.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:39 smithi191 ceph-mon[33016]: pgmap v440: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:41.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:41 smithi053 ceph-mon[29182]: pgmap v441: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:41.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:53:41.938+0000 7f074cff9640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f07380034c0 con 0x7f0758102a80 2024-04-28T12:53:41.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:53:41.938+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f07380034c0 con 0x7f0758102a80 2024-04-28T12:53:41.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:41 smithi077 ceph-mon[32553]: pgmap v441: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:42.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:41 smithi191 ceph-mon[33016]: pgmap v441: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:43.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:43 smithi053 ceph-mon[29182]: pgmap v442: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:43.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:43 smithi077 ceph-mon[32553]: pgmap v442: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:44.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:43 smithi191 ceph-mon[33016]: pgmap v442: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:45.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:45 smithi053 ceph-mon[29182]: pgmap v443: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:45.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:45 smithi077 ceph-mon[32553]: pgmap v443: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:46.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:45 smithi191 ceph-mon[33016]: pgmap v443: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:47.896 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:47 smithi053 ceph-mon[29182]: pgmap v444: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:47.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:47 smithi077 ceph-mon[32553]: pgmap v444: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:48.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:47 smithi191 ceph-mon[33016]: pgmap v444: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:49.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:49 smithi053 ceph-mon[29182]: pgmap v445: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:49.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:49 smithi077 ceph-mon[32553]: pgmap v445: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:50.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:49 smithi191 ceph-mon[33016]: pgmap v445: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:51.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:51 smithi053 ceph-mon[29182]: pgmap v446: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:51.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:51 smithi077 ceph-mon[32553]: pgmap v446: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:52.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:51 smithi191 ceph-mon[33016]: pgmap v446: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:53:53.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:53 smithi053 ceph-mon[29182]: pgmap v447: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:53.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:53 smithi077 ceph-mon[32553]: pgmap v447: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:54.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:53 smithi191 ceph-mon[33016]: pgmap v447: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:55.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:55 smithi053 ceph-mon[29182]: pgmap v448: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:55.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:55 smithi077 ceph-mon[32553]: pgmap v448: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:56.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:55 smithi191 ceph-mon[33016]: pgmap v448: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:57.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:57 smithi053 ceph-mon[29182]: pgmap v449: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:57.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:57 smithi077 ceph-mon[32553]: pgmap v449: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:58.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:57 smithi191 ceph-mon[33016]: pgmap v449: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:59.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:53:59 smithi053 ceph-mon[29182]: pgmap v450: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:59.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:53:59 smithi077 ceph-mon[32553]: pgmap v450: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:00.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:53:59 smithi191 ceph-mon[33016]: pgmap v450: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:01.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:01 smithi077 ceph-mon[32553]: pgmap v451: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:02.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:01 smithi191 ceph-mon[33016]: pgmap v451: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:02.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:01 smithi053 ceph-mon[29182]: pgmap v451: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:03.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:03 smithi077 ceph-mon[32553]: pgmap v452: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:04.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:03 smithi191 ceph-mon[33016]: pgmap v452: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:04.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:03 smithi053 ceph-mon[29182]: pgmap v452: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:05.949 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:05 smithi053 ceph-mon[29182]: pgmap v453: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:05.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:05 smithi077 ceph-mon[32553]: pgmap v453: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:06.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:05 smithi191 ceph-mon[33016]: pgmap v453: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:07.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:07 smithi077 ceph-mon[32553]: pgmap v454: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:07.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:07 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:54:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:07 smithi191 ceph-mon[33016]: pgmap v454: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:08.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:07 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:54:08.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:07 smithi053 ceph-mon[29182]: pgmap v454: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:08.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:07 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:54:08.949 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:54:08.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:54:08.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:54:09.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:54:09.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:54:09.039 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:54:09.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:54:09.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:54:09.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:54:10.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:09 smithi053 ceph-mon[29182]: pgmap v455: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:10.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:09 smithi077 ceph-mon[32553]: pgmap v455: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:10.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:09 smithi191 ceph-mon[33016]: pgmap v455: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:12.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:11 smithi053 ceph-mon[29182]: pgmap v456: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:12.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:11 smithi077 ceph-mon[32553]: pgmap v456: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:12.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:11 smithi191 ceph-mon[33016]: pgmap v456: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:14.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:13 smithi077 ceph-mon[32553]: pgmap v457: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:14.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:13 smithi191 ceph-mon[33016]: pgmap v457: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:14.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:13 smithi053 ceph-mon[29182]: pgmap v457: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:16.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:15 smithi077 ceph-mon[32553]: pgmap v458: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:16.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:15 smithi191 ceph-mon[33016]: pgmap v458: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:16.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:15 smithi053 ceph-mon[29182]: pgmap v458: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:18.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:17 smithi077 ceph-mon[32553]: pgmap v459: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:18.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:17 smithi191 ceph-mon[33016]: pgmap v459: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:18.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:17 smithi053 ceph-mon[29182]: pgmap v459: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:20.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:19 smithi077 ceph-mon[32553]: pgmap v460: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:20.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:19 smithi191 ceph-mon[33016]: pgmap v460: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:20.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:19 smithi053 ceph-mon[29182]: pgmap v460: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:22.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:21 smithi077 ceph-mon[32553]: pgmap v461: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:22.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:21 smithi191 ceph-mon[33016]: pgmap v461: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:22.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:21 smithi053 ceph-mon[29182]: pgmap v461: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:24.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:23 smithi077 ceph-mon[32553]: pgmap v462: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:24.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:23 smithi191 ceph-mon[33016]: pgmap v462: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:24.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:23 smithi053 ceph-mon[29182]: pgmap v462: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:26.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:25 smithi077 ceph-mon[32553]: pgmap v463: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:26.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:25 smithi191 ceph-mon[33016]: pgmap v463: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:26.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:25 smithi053 ceph-mon[29182]: pgmap v463: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:28.199 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:27 smithi077 ceph-mon[32553]: pgmap v464: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:28.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:27 smithi191 ceph-mon[33016]: pgmap v464: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:28.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:27 smithi053 ceph-mon[29182]: pgmap v464: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:30.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:29 smithi077 ceph-mon[32553]: pgmap v465: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:30.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:29 smithi191 ceph-mon[33016]: pgmap v465: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:30.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:29 smithi053 ceph-mon[29182]: pgmap v465: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:32.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:31 smithi077 ceph-mon[32553]: pgmap v466: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:31 smithi191 ceph-mon[33016]: pgmap v466: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:32.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:31 smithi053 ceph-mon[29182]: pgmap v466: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:34.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:33 smithi077 ceph-mon[32553]: pgmap v467: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:34.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:33 smithi191 ceph-mon[33016]: pgmap v467: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:34.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:33 smithi053 ceph-mon[29182]: pgmap v467: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:36.199 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:35 smithi053 ceph-mon[29182]: pgmap v468: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:36.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:35 smithi077 ceph-mon[32553]: pgmap v468: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:36.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:35 smithi191 ceph-mon[33016]: pgmap v468: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:38.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:37 smithi077 ceph-mon[32553]: pgmap v469: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:38.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:37 smithi191 ceph-mon[33016]: pgmap v469: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:38.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:37 smithi053 ceph-mon[29182]: pgmap v469: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:40.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:39 smithi077 ceph-mon[32553]: pgmap v470: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:40.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:39 smithi191 ceph-mon[33016]: pgmap v470: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:40.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:39 smithi053 ceph-mon[29182]: pgmap v470: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:42.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:41 smithi077 ceph-mon[32553]: pgmap v471: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:42.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:41 smithi191 ceph-mon[33016]: pgmap v471: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:42.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:41 smithi053 ceph-mon[29182]: pgmap v471: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:44.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:43 smithi077 ceph-mon[32553]: pgmap v472: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:44.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:43 smithi191 ceph-mon[33016]: pgmap v472: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:43 smithi053 ceph-mon[29182]: pgmap v472: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:46.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:45 smithi077 ceph-mon[32553]: pgmap v473: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:46.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:45 smithi191 ceph-mon[33016]: pgmap v473: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:46.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:45 smithi053 ceph-mon[29182]: pgmap v473: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:48.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:47 smithi077 ceph-mon[32553]: pgmap v474: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:48.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:47 smithi191 ceph-mon[33016]: pgmap v474: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:48.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:47 smithi053 ceph-mon[29182]: pgmap v474: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:50.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:49 smithi191 ceph-mon[33016]: pgmap v475: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:50.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:49 smithi053 ceph-mon[29182]: pgmap v475: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:50.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:49 smithi077 ceph-mon[32553]: pgmap v475: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:52.251 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:51 smithi077 ceph-mon[32553]: pgmap v476: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:52.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:51 smithi191 ceph-mon[33016]: pgmap v476: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:52.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:51 smithi053 ceph-mon[29182]: pgmap v476: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:54:54.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:53 smithi191 ceph-mon[33016]: pgmap v477: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:54.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:53 smithi053 ceph-mon[29182]: pgmap v477: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:54.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:53 smithi077 ceph-mon[32553]: pgmap v477: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:56.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:55 smithi191 ceph-mon[33016]: pgmap v478: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:56.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:55 smithi053 ceph-mon[29182]: pgmap v478: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:56.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:55 smithi077 ceph-mon[32553]: pgmap v478: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:58.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:57 smithi191 ceph-mon[33016]: pgmap v479: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:58.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:57 smithi053 ceph-mon[29182]: pgmap v479: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:58.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:57 smithi077 ceph-mon[32553]: pgmap v479: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:00.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:54:59 smithi191 ceph-mon[33016]: pgmap v480: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:00.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:54:59 smithi053 ceph-mon[29182]: pgmap v480: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:00.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:54:59 smithi077 ceph-mon[32553]: pgmap v480: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:02.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:01 smithi191 ceph-mon[33016]: pgmap v481: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:02.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:01 smithi053 ceph-mon[29182]: pgmap v481: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:02.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:01 smithi077 ceph-mon[32553]: pgmap v481: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:04.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:03 smithi191 ceph-mon[33016]: pgmap v482: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:04.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:03 smithi053 ceph-mon[29182]: pgmap v482: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:04.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:03 smithi077 ceph-mon[32553]: pgmap v482: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:06.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:05 smithi191 ceph-mon[33016]: pgmap v483: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:06.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:05 smithi053 ceph-mon[29182]: pgmap v483: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:06.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:05 smithi077 ceph-mon[32553]: pgmap v483: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:08.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:07 smithi191 ceph-mon[33016]: pgmap v484: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:08.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:07 smithi053 ceph-mon[29182]: pgmap v484: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:08.449 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:07 smithi077 ceph-mon[32553]: pgmap v484: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:55:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:55:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:55:09.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:08 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:55:09.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:55:09.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:55:09.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:55:09.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:08 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:55:09.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:55:09.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:55:09.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:55:09.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:08 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:55:10.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:09 smithi191 ceph-mon[33016]: pgmap v485: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:10.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:09 smithi053 ceph-mon[29182]: pgmap v485: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:10.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:09 smithi077 ceph-mon[32553]: pgmap v485: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:12.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:11 smithi191 ceph-mon[33016]: pgmap v486: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:12.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:11 smithi053 ceph-mon[29182]: pgmap v486: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:12.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:11 smithi077 ceph-mon[32553]: pgmap v486: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:14.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:13 smithi191 ceph-mon[33016]: pgmap v487: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:14.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:13 smithi053 ceph-mon[29182]: pgmap v487: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:14.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:13 smithi077 ceph-mon[32553]: pgmap v487: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:16.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:15 smithi191 ceph-mon[33016]: pgmap v488: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:16.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:15 smithi053 ceph-mon[29182]: pgmap v488: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:16.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:15 smithi077 ceph-mon[32553]: pgmap v488: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:18.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:18 smithi191 ceph-mon[33016]: pgmap v489: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:18.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:18 smithi053 ceph-mon[29182]: pgmap v489: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:18.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:18 smithi077 ceph-mon[32553]: pgmap v489: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:20.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:20 smithi191 ceph-mon[33016]: pgmap v490: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:20.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:20 smithi053 ceph-mon[29182]: pgmap v490: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:20.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:20 smithi077 ceph-mon[32553]: pgmap v490: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:22.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:22 smithi191 ceph-mon[33016]: pgmap v491: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:22.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:22 smithi053 ceph-mon[29182]: pgmap v491: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:22.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:22 smithi077 ceph-mon[32553]: pgmap v491: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:24.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:24 smithi191 ceph-mon[33016]: pgmap v492: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:24.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:24 smithi053 ceph-mon[29182]: pgmap v492: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:24.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:24 smithi077 ceph-mon[32553]: pgmap v492: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:26.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:26 smithi191 ceph-mon[33016]: pgmap v493: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:26.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:26 smithi053 ceph-mon[29182]: pgmap v493: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:26.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:26 smithi077 ceph-mon[32553]: pgmap v493: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:28.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:28 smithi191 ceph-mon[33016]: pgmap v494: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:28.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:28 smithi053 ceph-mon[29182]: pgmap v494: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:28.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:28 smithi077 ceph-mon[32553]: pgmap v494: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:30.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:30 smithi191 ceph-mon[33016]: pgmap v495: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:30.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:30 smithi053 ceph-mon[29182]: pgmap v495: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:30.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:30 smithi077 ceph-mon[32553]: pgmap v495: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:32 smithi191 ceph-mon[33016]: pgmap v496: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:32 smithi053 ceph-mon[29182]: pgmap v496: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:32.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:32 smithi077 ceph-mon[32553]: pgmap v496: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:34.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:34 smithi191 ceph-mon[33016]: pgmap v497: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:34.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:34 smithi053 ceph-mon[29182]: pgmap v497: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:34.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:34 smithi077 ceph-mon[32553]: pgmap v497: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:36.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:36 smithi191 ceph-mon[33016]: pgmap v498: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:36.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:36 smithi053 ceph-mon[29182]: pgmap v498: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:36 smithi077 ceph-mon[32553]: pgmap v498: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:38.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:38 smithi191 ceph-mon[33016]: pgmap v499: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:38.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:38 smithi053 ceph-mon[29182]: pgmap v499: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:38.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:38 smithi077 ceph-mon[32553]: pgmap v499: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:40.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:40 smithi191 ceph-mon[33016]: pgmap v500: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:40.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:40 smithi053 ceph-mon[29182]: pgmap v500: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:40.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:40 smithi077 ceph-mon[32553]: pgmap v500: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:42.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:42 smithi191 ceph-mon[33016]: pgmap v501: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:42.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:42 smithi053 ceph-mon[29182]: pgmap v501: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:42.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:42 smithi077 ceph-mon[32553]: pgmap v501: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:44.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:44 smithi191 ceph-mon[33016]: pgmap v502: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:44 smithi053 ceph-mon[29182]: pgmap v502: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:44.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:44 smithi077 ceph-mon[32553]: pgmap v502: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:46.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:46 smithi053 ceph-mon[29182]: pgmap v503: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:46.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:46 smithi077 ceph-mon[32553]: pgmap v503: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:46.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:46 smithi191 ceph-mon[33016]: pgmap v503: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:48.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:48 smithi053 ceph-mon[29182]: pgmap v504: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:48.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:48 smithi077 ceph-mon[32553]: pgmap v504: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:48.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:48 smithi191 ceph-mon[33016]: pgmap v504: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:50.342 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:50 smithi191 ceph-mon[33016]: pgmap v505: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:50.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:50 smithi053 ceph-mon[29182]: pgmap v505: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:50.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:50 smithi077 ceph-mon[32553]: pgmap v505: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:52.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:52 smithi053 ceph-mon[29182]: pgmap v506: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:52.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:52 smithi077 ceph-mon[32553]: pgmap v506: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:52.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:52 smithi191 ceph-mon[33016]: pgmap v506: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:55:54.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:54 smithi053 ceph-mon[29182]: pgmap v507: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:54.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:54 smithi077 ceph-mon[32553]: pgmap v507: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:54.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:54 smithi191 ceph-mon[33016]: pgmap v507: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:56.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:56 smithi053 ceph-mon[29182]: pgmap v508: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:56.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:56 smithi077 ceph-mon[32553]: pgmap v508: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:56.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:56 smithi191 ceph-mon[33016]: pgmap v508: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:58.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:55:58 smithi053 ceph-mon[29182]: pgmap v509: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:58.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:55:58 smithi077 ceph-mon[32553]: pgmap v509: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:58.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:55:58 smithi191 ceph-mon[33016]: pgmap v509: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:00.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:00 smithi053 ceph-mon[29182]: pgmap v510: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:00.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:00 smithi077 ceph-mon[32553]: pgmap v510: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:00.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:00 smithi191 ceph-mon[33016]: pgmap v510: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:02.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:02 smithi053 ceph-mon[29182]: pgmap v511: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:02.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:02 smithi077 ceph-mon[32553]: pgmap v511: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:02.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:02 smithi191 ceph-mon[33016]: pgmap v511: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:04.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:04 smithi053 ceph-mon[29182]: pgmap v512: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:04.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:04 smithi077 ceph-mon[32553]: pgmap v512: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:04.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:04 smithi191 ceph-mon[33016]: pgmap v512: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:06.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:06 smithi053 ceph-mon[29182]: pgmap v513: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:06.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:06 smithi077 ceph-mon[32553]: pgmap v513: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:06.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:06 smithi191 ceph-mon[33016]: pgmap v513: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:08.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:08 smithi053 ceph-mon[29182]: pgmap v514: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:08.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:08 smithi077 ceph-mon[32553]: pgmap v514: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:08 smithi191 ceph-mon[33016]: pgmap v514: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:09.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:09 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:56:09.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:09 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:56:09.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:09 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:56:10.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:10 smithi053 ceph-mon[29182]: pgmap v515: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:10.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:56:10.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:56:10.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:56:10.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:10 smithi077 ceph-mon[32553]: pgmap v515: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:10.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:56:10.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:56:10.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:56:10.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:10 smithi191 ceph-mon[33016]: pgmap v515: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:10.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:56:10.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:56:10.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:56:12.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:12 smithi053 ceph-mon[29182]: pgmap v516: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:12.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:12 smithi077 ceph-mon[32553]: pgmap v516: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:12.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:12 smithi191 ceph-mon[33016]: pgmap v516: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:14.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:14 smithi053 ceph-mon[29182]: pgmap v517: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:14.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:14 smithi077 ceph-mon[32553]: pgmap v517: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:14.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:14 smithi191 ceph-mon[33016]: pgmap v517: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:16.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:16 smithi053 ceph-mon[29182]: pgmap v518: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:16.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:16 smithi077 ceph-mon[32553]: pgmap v518: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:16.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:16 smithi191 ceph-mon[33016]: pgmap v518: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:18.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:18 smithi053 ceph-mon[29182]: pgmap v519: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:18.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:18 smithi077 ceph-mon[32553]: pgmap v519: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:18.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:18 smithi191 ceph-mon[33016]: pgmap v519: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:20.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:20 smithi053 ceph-mon[29182]: pgmap v520: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:20.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:20 smithi077 ceph-mon[32553]: pgmap v520: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:20.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:20 smithi191 ceph-mon[33016]: pgmap v520: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:22.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:22 smithi053 ceph-mon[29182]: pgmap v521: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:22.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:22 smithi077 ceph-mon[32553]: pgmap v521: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:22.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:22 smithi191 ceph-mon[33016]: pgmap v521: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:24.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:24 smithi053 ceph-mon[29182]: pgmap v522: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:24.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:24 smithi077 ceph-mon[32553]: pgmap v522: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:24.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:24 smithi191 ceph-mon[33016]: pgmap v522: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:26.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:26 smithi053 ceph-mon[29182]: pgmap v523: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:26.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:26 smithi077 ceph-mon[32553]: pgmap v523: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:26.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:26 smithi191 ceph-mon[33016]: pgmap v523: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:28.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:28 smithi053 ceph-mon[29182]: pgmap v524: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:28.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:28 smithi077 ceph-mon[32553]: pgmap v524: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:28.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:28 smithi191 ceph-mon[33016]: pgmap v524: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:30.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:30 smithi053 ceph-mon[29182]: pgmap v525: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:30.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:30 smithi077 ceph-mon[32553]: pgmap v525: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:30.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:30 smithi191 ceph-mon[33016]: pgmap v525: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:32.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:32 smithi053 ceph-mon[29182]: pgmap v526: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:32.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:32 smithi077 ceph-mon[32553]: pgmap v526: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:32.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:32 smithi191 ceph-mon[33016]: pgmap v526: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:34.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:34 smithi053 ceph-mon[29182]: pgmap v527: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:34.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:34 smithi077 ceph-mon[32553]: pgmap v527: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:34.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:34 smithi191 ceph-mon[33016]: pgmap v527: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:36.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:36 smithi053 ceph-mon[29182]: pgmap v528: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:36.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:36 smithi077 ceph-mon[32553]: pgmap v528: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:36.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:36 smithi191 ceph-mon[33016]: pgmap v528: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:38.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:38 smithi053 ceph-mon[29182]: pgmap v529: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:38.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:38 smithi077 ceph-mon[32553]: pgmap v529: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:38.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:38 smithi191 ceph-mon[33016]: pgmap v529: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:40.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:40 smithi053 ceph-mon[29182]: pgmap v530: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:40.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:40 smithi077 ceph-mon[32553]: pgmap v530: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:40.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:40 smithi191 ceph-mon[33016]: pgmap v530: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:41.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:56:41.965+0000 7f074cff9640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f0738005850 con 0x7f0758102a80 2024-04-28T12:56:41.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:56:41.965+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f0738005850 con 0x7f0758102a80 2024-04-28T12:56:42.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:42 smithi053 ceph-mon[29182]: pgmap v531: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:42.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:42 smithi077 ceph-mon[32553]: pgmap v531: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:42.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:42 smithi191 ceph-mon[33016]: pgmap v531: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:44.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:44 smithi053 ceph-mon[29182]: pgmap v532: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:44.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:44 smithi077 ceph-mon[32553]: pgmap v532: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:44.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:44 smithi191 ceph-mon[33016]: pgmap v532: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:46.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:46 smithi053 ceph-mon[29182]: pgmap v533: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:46.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:46 smithi077 ceph-mon[32553]: pgmap v533: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:46.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:46 smithi191 ceph-mon[33016]: pgmap v533: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:48.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:48 smithi053 ceph-mon[29182]: pgmap v534: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:48.453 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:48 smithi077 ceph-mon[32553]: pgmap v534: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:48.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:48 smithi191 ceph-mon[33016]: pgmap v534: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:50.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:50 smithi053 ceph-mon[29182]: pgmap v535: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:50.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:50 smithi077 ceph-mon[32553]: pgmap v535: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:50.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:50 smithi191 ceph-mon[33016]: pgmap v535: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:52.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:52 smithi077 ceph-mon[32553]: pgmap v536: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:52.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:52 smithi191 ceph-mon[33016]: pgmap v536: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:52.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:52 smithi053 ceph-mon[29182]: pgmap v536: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:56:54.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:54 smithi077 ceph-mon[32553]: pgmap v537: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:54.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:54 smithi191 ceph-mon[33016]: pgmap v537: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:54.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:54 smithi053 ceph-mon[29182]: pgmap v537: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:56.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:56 smithi077 ceph-mon[32553]: pgmap v538: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:56.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:56 smithi191 ceph-mon[33016]: pgmap v538: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:56.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:56 smithi053 ceph-mon[29182]: pgmap v538: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:58.453 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:56:58 smithi077 ceph-mon[32553]: pgmap v539: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:58.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:56:58 smithi191 ceph-mon[33016]: pgmap v539: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:58.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:56:58 smithi053 ceph-mon[29182]: pgmap v539: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:00.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:00 smithi077 ceph-mon[32553]: pgmap v540: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:00.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:00 smithi191 ceph-mon[33016]: pgmap v540: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:00.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:00 smithi053 ceph-mon[29182]: pgmap v540: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:02.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:02 smithi077 ceph-mon[32553]: pgmap v541: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:02.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:02 smithi191 ceph-mon[33016]: pgmap v541: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:02.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:02 smithi053 ceph-mon[29182]: pgmap v541: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:04.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:04 smithi077 ceph-mon[32553]: pgmap v542: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:04.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:04 smithi191 ceph-mon[33016]: pgmap v542: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:04.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:04 smithi053 ceph-mon[29182]: pgmap v542: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:06.450 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:06 smithi053 ceph-mon[29182]: pgmap v543: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:06.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:06 smithi077 ceph-mon[32553]: pgmap v543: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:06.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:06 smithi191 ceph-mon[33016]: pgmap v543: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:08.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:08 smithi077 ceph-mon[32553]: pgmap v544: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:08.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:08 smithi191 ceph-mon[33016]: pgmap v544: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:08.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:08 smithi053 ceph-mon[29182]: pgmap v544: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:10.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:10 smithi077 ceph-mon[32553]: pgmap v545: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:10.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:57:10.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:10 smithi191 ceph-mon[33016]: pgmap v545: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:10.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:57:10.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:10 smithi053 ceph-mon[29182]: pgmap v545: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:10.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:57:11.450 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:57:11.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:57:11.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:57:11.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi053 to 177.2M 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: Unable to set osd_memory_target on smithi053 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:57:11.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:57:12.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:12 smithi053 ceph-mon[29182]: pgmap v546: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:12.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:12 smithi077 ceph-mon[32553]: pgmap v546: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:12.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:12 smithi191 ceph-mon[33016]: pgmap v546: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:14.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:14 smithi053 ceph-mon[29182]: pgmap v547: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:14.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:14 smithi077 ceph-mon[32553]: pgmap v547: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:14.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:14 smithi191 ceph-mon[33016]: pgmap v547: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:16.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:16 smithi053 ceph-mon[29182]: pgmap v548: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:16.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:16 smithi077 ceph-mon[32553]: pgmap v548: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:16.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:16 smithi191 ceph-mon[33016]: pgmap v548: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:18.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:18 smithi053 ceph-mon[29182]: pgmap v549: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:18.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:18 smithi077 ceph-mon[32553]: pgmap v549: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:18.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:18 smithi191 ceph-mon[33016]: pgmap v549: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:20.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:20 smithi053 ceph-mon[29182]: pgmap v550: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:20.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:20 smithi077 ceph-mon[32553]: pgmap v550: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:20.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:20 smithi191 ceph-mon[33016]: pgmap v550: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:22.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:22 smithi053 ceph-mon[29182]: pgmap v551: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:22.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:22 smithi077 ceph-mon[32553]: pgmap v551: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:22.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:22 smithi191 ceph-mon[33016]: pgmap v551: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:24.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:24 smithi053 ceph-mon[29182]: pgmap v552: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:24.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:24 smithi077 ceph-mon[32553]: pgmap v552: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:24.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:24 smithi191 ceph-mon[33016]: pgmap v552: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:26.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:26 smithi053 ceph-mon[29182]: pgmap v553: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:26.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:26 smithi077 ceph-mon[32553]: pgmap v553: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:26.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:26 smithi191 ceph-mon[33016]: pgmap v553: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:28.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:28 smithi053 ceph-mon[29182]: pgmap v554: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:28.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:28 smithi077 ceph-mon[32553]: pgmap v554: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:28.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:28 smithi191 ceph-mon[33016]: pgmap v554: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:30.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:30 smithi053 ceph-mon[29182]: pgmap v555: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:30.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:30 smithi077 ceph-mon[32553]: pgmap v555: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:30.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:30 smithi191 ceph-mon[33016]: pgmap v555: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:32.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:32 smithi053 ceph-mon[29182]: pgmap v556: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:32.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:32 smithi077 ceph-mon[32553]: pgmap v556: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:32.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:32 smithi191 ceph-mon[33016]: pgmap v556: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:34.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:34 smithi053 ceph-mon[29182]: pgmap v557: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:34.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:34 smithi077 ceph-mon[32553]: pgmap v557: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:34.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:34 smithi191 ceph-mon[33016]: pgmap v557: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:36.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:36 smithi053 ceph-mon[29182]: pgmap v558: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:36.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:36 smithi077 ceph-mon[32553]: pgmap v558: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:36.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:36 smithi191 ceph-mon[33016]: pgmap v558: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:38.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:38 smithi053 ceph-mon[29182]: pgmap v559: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:38.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:38 smithi077 ceph-mon[32553]: pgmap v559: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:38.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:38 smithi191 ceph-mon[33016]: pgmap v559: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:40.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:40 smithi053 ceph-mon[29182]: pgmap v560: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:40.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:40 smithi077 ceph-mon[32553]: pgmap v560: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:40.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:40 smithi191 ceph-mon[33016]: pgmap v560: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:42.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:42 smithi053 ceph-mon[29182]: pgmap v561: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:42.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:42 smithi077 ceph-mon[32553]: pgmap v561: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:42 smithi191 ceph-mon[33016]: pgmap v561: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:44.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:44 smithi053 ceph-mon[29182]: pgmap v562: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:44.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:44 smithi077 ceph-mon[32553]: pgmap v562: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:44.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:44 smithi191 ceph-mon[33016]: pgmap v562: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:46.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:46 smithi053 ceph-mon[29182]: pgmap v563: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:46.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:46 smithi077 ceph-mon[32553]: pgmap v563: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:46.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:46 smithi191 ceph-mon[33016]: pgmap v563: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:48.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:48 smithi053 ceph-mon[29182]: pgmap v564: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:48.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:48 smithi077 ceph-mon[32553]: pgmap v564: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:48.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:48 smithi191 ceph-mon[33016]: pgmap v564: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:50.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:50 smithi053 ceph-mon[29182]: pgmap v565: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:50.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:50 smithi077 ceph-mon[32553]: pgmap v565: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:50.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:50 smithi191 ceph-mon[33016]: pgmap v565: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:52.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:52 smithi053 ceph-mon[29182]: pgmap v566: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:52.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:52 smithi077 ceph-mon[32553]: pgmap v566: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:52.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:52 smithi191 ceph-mon[33016]: pgmap v566: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:57:54.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:54 smithi053 ceph-mon[29182]: pgmap v567: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:54.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:54 smithi077 ceph-mon[32553]: pgmap v567: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:54.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:54 smithi191 ceph-mon[33016]: pgmap v567: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:56.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:56 smithi053 ceph-mon[29182]: pgmap v568: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:56.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:56 smithi077 ceph-mon[32553]: pgmap v568: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:56.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:56 smithi191 ceph-mon[33016]: pgmap v568: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:58.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:57:58 smithi053 ceph-mon[29182]: pgmap v569: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:58.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:57:58 smithi077 ceph-mon[32553]: pgmap v569: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:58.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:57:58 smithi191 ceph-mon[33016]: pgmap v569: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:00.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:00 smithi077 ceph-mon[32553]: pgmap v570: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:00.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:00 smithi191 ceph-mon[33016]: pgmap v570: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:00.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:00 smithi053 ceph-mon[29182]: pgmap v570: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:02.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:02 smithi077 ceph-mon[32553]: pgmap v571: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:02.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:02 smithi191 ceph-mon[33016]: pgmap v571: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:02.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:02 smithi053 ceph-mon[29182]: pgmap v571: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:04.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:04 smithi077 ceph-mon[32553]: pgmap v572: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:04.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:04 smithi191 ceph-mon[33016]: pgmap v572: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:04.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:04 smithi053 ceph-mon[29182]: pgmap v572: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:06.700 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:06 smithi053 ceph-mon[29182]: pgmap v573: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:06.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:06 smithi077 ceph-mon[32553]: pgmap v573: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:06.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:06 smithi191 ceph-mon[33016]: pgmap v573: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:08.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:08 smithi077 ceph-mon[32553]: pgmap v574: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:08.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:08 smithi191 ceph-mon[33016]: pgmap v574: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:08.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:08 smithi053 ceph-mon[29182]: pgmap v574: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:09.700 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:09 smithi077 ceph-mon[32553]: pgmap v575: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:09.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:09 smithi191 ceph-mon[33016]: pgmap v575: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:09.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:09 smithi053 ceph-mon[29182]: pgmap v575: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:10.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:10 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:10 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:58:10.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:10 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:58:11.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: pgmap v576: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:11.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: Unable to set osd_memory_target on smithi077 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: Unable to set osd_memory_target on smithi191 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:58:11.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: pgmap v576: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-28T12:58:11.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-28T12:58:11.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: Unable to set osd_memory_target on smithi077 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-28T12:58:11.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: Unable to set osd_memory_target on smithi191 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-28T12:58:11.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:58:11.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:58:11.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: pgmap v576: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: Unable to set osd_memory_target on smithi077 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-28T12:58:11.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: Unable to set osd_memory_target on smithi191 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-28T12:58:11.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:58:11.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:58:11.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:58:13.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:13 smithi077 ceph-mon[32553]: pgmap v577: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:14.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:13 smithi191 ceph-mon[33016]: pgmap v577: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:14.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:13 smithi053 ceph-mon[29182]: pgmap v577: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:15.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:15 smithi077 ceph-mon[32553]: pgmap v578: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:16.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:15 smithi191 ceph-mon[33016]: pgmap v578: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:16.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:15 smithi053 ceph-mon[29182]: pgmap v578: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:17.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:17 smithi077 ceph-mon[32553]: pgmap v579: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:18.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:17 smithi191 ceph-mon[33016]: pgmap v579: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:18.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:17 smithi053 ceph-mon[29182]: pgmap v579: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:19.950 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:19 smithi077 ceph-mon[32553]: pgmap v580: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:20.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:19 smithi191 ceph-mon[33016]: pgmap v580: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:20.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:19 smithi053 ceph-mon[29182]: pgmap v580: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:21.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:21 smithi077 ceph-mon[32553]: pgmap v581: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:22.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:21 smithi191 ceph-mon[33016]: pgmap v581: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:22.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:21 smithi053 ceph-mon[29182]: pgmap v581: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:23.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:23 smithi077 ceph-mon[32553]: pgmap v582: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:24.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:23 smithi191 ceph-mon[33016]: pgmap v582: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:24.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:23 smithi053 ceph-mon[29182]: pgmap v582: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:25.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:25 smithi077 ceph-mon[32553]: pgmap v583: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:26.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:25 smithi191 ceph-mon[33016]: pgmap v583: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:26.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:25 smithi053 ceph-mon[29182]: pgmap v583: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:27.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:27 smithi077 ceph-mon[32553]: pgmap v584: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:28.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:27 smithi191 ceph-mon[33016]: pgmap v584: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:28.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:27 smithi053 ceph-mon[29182]: pgmap v584: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:29.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:29 smithi077 ceph-mon[32553]: pgmap v585: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:30.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:29 smithi191 ceph-mon[33016]: pgmap v585: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:30.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:29 smithi053 ceph-mon[29182]: pgmap v585: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:31.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:31 smithi077 ceph-mon[32553]: pgmap v586: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:32.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:31 smithi191 ceph-mon[33016]: pgmap v586: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:32.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:31 smithi053 ceph-mon[29182]: pgmap v586: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:33.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:33 smithi077 ceph-mon[32553]: pgmap v587: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:34.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:33 smithi191 ceph-mon[33016]: pgmap v587: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:34.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:33 smithi053 ceph-mon[29182]: pgmap v587: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:35.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:35 smithi077 ceph-mon[32553]: pgmap v588: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:36.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:35 smithi191 ceph-mon[33016]: pgmap v588: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:36.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:35 smithi053 ceph-mon[29182]: pgmap v588: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:38.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:37 smithi191 ceph-mon[33016]: pgmap v589: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:38.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:37 smithi053 ceph-mon[29182]: pgmap v589: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:38.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:37 smithi077 ceph-mon[32553]: pgmap v589: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:40.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:39 smithi191 ceph-mon[33016]: pgmap v590: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:40.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:39 smithi053 ceph-mon[29182]: pgmap v590: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:40.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:39 smithi077 ceph-mon[32553]: pgmap v590: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:42.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:41 smithi191 ceph-mon[33016]: pgmap v591: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:42.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:41 smithi053 ceph-mon[29182]: pgmap v591: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:42.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:41 smithi077 ceph-mon[32553]: pgmap v591: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:44.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:43 smithi191 ceph-mon[33016]: pgmap v592: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:44.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:43 smithi053 ceph-mon[29182]: pgmap v592: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:44.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:43 smithi077 ceph-mon[32553]: pgmap v592: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:46.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:45 smithi191 ceph-mon[33016]: pgmap v593: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:46.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:45 smithi053 ceph-mon[29182]: pgmap v593: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:46.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:45 smithi077 ceph-mon[32553]: pgmap v593: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:48.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:47 smithi191 ceph-mon[33016]: pgmap v594: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:48.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:47 smithi053 ceph-mon[29182]: pgmap v594: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:47 smithi077 ceph-mon[32553]: pgmap v594: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:50.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:49 smithi191 ceph-mon[33016]: pgmap v595: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:50.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:49 smithi053 ceph-mon[29182]: pgmap v595: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:50.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:49 smithi077 ceph-mon[32553]: pgmap v595: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:52.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:51 smithi191 ceph-mon[33016]: pgmap v596: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:52.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:51 smithi053 ceph-mon[29182]: pgmap v596: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:52.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:51 smithi077 ceph-mon[32553]: pgmap v596: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:58:54.003 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:53 smithi077 ceph-mon[32553]: pgmap v597: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:54.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:53 smithi191 ceph-mon[33016]: pgmap v597: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:54.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:53 smithi053 ceph-mon[29182]: pgmap v597: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:56.041 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:55 smithi191 ceph-mon[33016]: pgmap v598: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:56.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:55 smithi053 ceph-mon[29182]: pgmap v598: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:56.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:55 smithi077 ceph-mon[32553]: pgmap v598: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:56.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:58:56.985+0000 7f074cff9640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f0738005b70 con 0x7f0758102a80 2024-04-28T12:58:56.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:58:56.985+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f0738005b70 con 0x7f0758102a80 2024-04-28T12:58:58.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:57 smithi191 ceph-mon[33016]: pgmap v599: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:58.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:57 smithi053 ceph-mon[29182]: pgmap v599: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:58.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:57 smithi077 ceph-mon[32553]: pgmap v599: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:00.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:58:59 smithi191 ceph-mon[33016]: pgmap v600: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:00.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:58:59 smithi053 ceph-mon[29182]: pgmap v600: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:00.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:58:59 smithi077 ceph-mon[32553]: pgmap v600: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:02.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:01 smithi191 ceph-mon[33016]: pgmap v601: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:02.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:01 smithi053 ceph-mon[29182]: pgmap v601: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:02.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:01 smithi077 ceph-mon[32553]: pgmap v601: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:04.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:03 smithi191 ceph-mon[33016]: pgmap v602: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:04.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:03 smithi053 ceph-mon[29182]: pgmap v602: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:04.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:03 smithi077 ceph-mon[32553]: pgmap v602: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:06.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:05 smithi191 ceph-mon[33016]: pgmap v603: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:06.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:05 smithi053 ceph-mon[29182]: pgmap v603: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:06.227 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:05 smithi077 ceph-mon[32553]: pgmap v603: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:08.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:07 smithi191 ceph-mon[33016]: pgmap v604: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:08.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:07 smithi053 ceph-mon[29182]: pgmap v604: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:08.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:07 smithi077 ceph-mon[32553]: pgmap v604: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:10.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:09 smithi191 ceph-mon[33016]: pgmap v605: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:10.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:09 smithi053 ceph-mon[29182]: pgmap v605: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:10.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:09 smithi077 ceph-mon[32553]: pgmap v605: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:12.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:11 smithi191 ceph-mon[33016]: pgmap v606: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:12.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:11 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:59:12.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:11 smithi053 ceph-mon[29182]: pgmap v606: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:12.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:11 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:59:12.200 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:11 smithi077 ceph-mon[32553]: pgmap v606: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:12.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:11 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:59:13.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:59:13.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:59:13.041 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:59:13.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:59:13.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:59:13.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:59:13.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:59:13.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:59:13.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T12:59:14.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:13 smithi053 ceph-mon[29182]: pgmap v607: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:14.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:13 smithi077 ceph-mon[32553]: pgmap v607: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:14.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:13 smithi191 ceph-mon[33016]: pgmap v607: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:16.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:15 smithi053 ceph-mon[29182]: pgmap v608: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:16.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:15 smithi077 ceph-mon[32553]: pgmap v608: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:16.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:15 smithi191 ceph-mon[33016]: pgmap v608: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:18.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:17 smithi077 ceph-mon[32553]: pgmap v609: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:18.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:17 smithi191 ceph-mon[33016]: pgmap v609: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:18.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:17 smithi053 ceph-mon[29182]: pgmap v609: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:20.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:19 smithi077 ceph-mon[32553]: pgmap v610: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:20.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:19 smithi191 ceph-mon[33016]: pgmap v610: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:20.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:19 smithi053 ceph-mon[29182]: pgmap v610: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:22.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:21 smithi077 ceph-mon[32553]: pgmap v611: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:22.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:21 smithi191 ceph-mon[33016]: pgmap v611: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:22.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:21 smithi053 ceph-mon[29182]: pgmap v611: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:24.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:23 smithi077 ceph-mon[32553]: pgmap v612: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:24.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:23 smithi191 ceph-mon[33016]: pgmap v612: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:24.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:23 smithi053 ceph-mon[29182]: pgmap v612: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:26.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:25 smithi077 ceph-mon[32553]: pgmap v613: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:26.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:25 smithi191 ceph-mon[33016]: pgmap v613: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:26.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:25 smithi053 ceph-mon[29182]: pgmap v613: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:28.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:27 smithi077 ceph-mon[32553]: pgmap v614: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:28.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:27 smithi191 ceph-mon[33016]: pgmap v614: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:28.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:27 smithi053 ceph-mon[29182]: pgmap v614: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:30.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:29 smithi077 ceph-mon[32553]: pgmap v615: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:30.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:29 smithi191 ceph-mon[33016]: pgmap v615: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:30.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:29 smithi053 ceph-mon[29182]: pgmap v615: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:32.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:31 smithi077 ceph-mon[32553]: pgmap v616: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:31 smithi191 ceph-mon[33016]: pgmap v616: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:32.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:31 smithi053 ceph-mon[29182]: pgmap v616: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:34.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:33 smithi077 ceph-mon[32553]: pgmap v617: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:59:34.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:33 smithi191 ceph-mon[33016]: pgmap v617: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:59:34.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:33 smithi053 ceph-mon[29182]: pgmap v617: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:59:36.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:35 smithi077 ceph-mon[32553]: pgmap v618: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:36.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:35 smithi191 ceph-mon[33016]: pgmap v618: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:36.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:35 smithi053 ceph-mon[29182]: pgmap v618: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:38.201 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:37 smithi053 ceph-mon[29182]: pgmap v619: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:59:38.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:37 smithi077 ceph-mon[32553]: pgmap v619: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:59:38.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:37 smithi191 ceph-mon[33016]: pgmap v619: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:59:40.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:39 smithi077 ceph-mon[32553]: pgmap v620: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:59:40.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:39 smithi191 ceph-mon[33016]: pgmap v620: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:59:40.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:39 smithi053 ceph-mon[29182]: pgmap v620: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:59:41.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:59:41.991+0000 7f074cff9640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f0738005ed0 con 0x7f0758102a80 2024-04-28T12:59:41.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T12:59:41.992+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f0738005ed0 con 0x7f0758102a80 2024-04-28T12:59:42.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:41 smithi077 ceph-mon[32553]: pgmap v621: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:42.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:41 smithi191 ceph-mon[33016]: pgmap v621: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:42.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:41 smithi053 ceph-mon[29182]: pgmap v621: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:44.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:43 smithi077 ceph-mon[32553]: pgmap v622: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:44.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:43 smithi191 ceph-mon[33016]: pgmap v622: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:43 smithi053 ceph-mon[29182]: pgmap v622: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:46.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:45 smithi077 ceph-mon[32553]: pgmap v623: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:46.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:45 smithi191 ceph-mon[33016]: pgmap v623: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:46.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:45 smithi053 ceph-mon[29182]: pgmap v623: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:47 smithi077 ceph-mon[32553]: pgmap v624: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:48.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:47 smithi191 ceph-mon[33016]: pgmap v624: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:48.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:47 smithi053 ceph-mon[29182]: pgmap v624: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:50.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:49 smithi077 ceph-mon[32553]: pgmap v625: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:50.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:49 smithi191 ceph-mon[33016]: pgmap v625: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:50.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:49 smithi053 ceph-mon[29182]: pgmap v625: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:52.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:51 smithi077 ceph-mon[32553]: pgmap v626: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:52.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:51 smithi191 ceph-mon[33016]: pgmap v626: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:52.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:51 smithi053 ceph-mon[29182]: pgmap v626: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:54.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:53 smithi191 ceph-mon[33016]: pgmap v627: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:54.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:53 smithi053 ceph-mon[29182]: pgmap v627: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:54.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:53 smithi077 ceph-mon[32553]: pgmap v627: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:56.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:55 smithi191 ceph-mon[33016]: pgmap v628: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:56.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:55 smithi053 ceph-mon[29182]: pgmap v628: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:56.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:55 smithi077 ceph-mon[32553]: pgmap v628: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:58.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:57 smithi191 ceph-mon[33016]: pgmap v629: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:58.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:57 smithi053 ceph-mon[29182]: pgmap v629: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:58.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:57 smithi077 ceph-mon[32553]: pgmap v629: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:00.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 12:59:59 smithi191 ceph-mon[33016]: pgmap v630: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:00.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 12:59:59 smithi053 ceph-mon[29182]: pgmap v630: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:00.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 12:59:59 smithi077 ceph-mon[32553]: pgmap v630: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:01.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:00 smithi191 ceph-mon[33016]: overall HEALTH_OK 2024-04-28T13:00:01.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:00 smithi053 ceph-mon[29182]: overall HEALTH_OK 2024-04-28T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:00 smithi077 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T13:00:02.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:01 smithi191 ceph-mon[33016]: pgmap v631: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:02.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:01 smithi053 ceph-mon[29182]: pgmap v631: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:02.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:01 smithi077 ceph-mon[32553]: pgmap v631: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:04.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:03 smithi191 ceph-mon[33016]: pgmap v632: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:04.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:03 smithi053 ceph-mon[29182]: pgmap v632: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:04.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:03 smithi077 ceph-mon[32553]: pgmap v632: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:06.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:05 smithi191 ceph-mon[33016]: pgmap v633: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:06.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:05 smithi053 ceph-mon[29182]: pgmap v633: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:06.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:05 smithi077 ceph-mon[32553]: pgmap v633: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:08.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:07 smithi191 ceph-mon[33016]: pgmap v634: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:08.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:07 smithi053 ceph-mon[29182]: pgmap v634: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:08.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:07 smithi077 ceph-mon[32553]: pgmap v634: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:10.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:09 smithi191 ceph-mon[33016]: pgmap v635: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:10.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:09 smithi053 ceph-mon[29182]: pgmap v635: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:10.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:09 smithi077 ceph-mon[32553]: pgmap v635: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:12.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:11 smithi191 ceph-mon[33016]: pgmap v636: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:12.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:11 smithi053 ceph-mon[29182]: pgmap v636: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:12.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:11 smithi077 ceph-mon[32553]: pgmap v636: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:13.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:00:13.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:00:13.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:00:13.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:12 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:00:13.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:00:13.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:00:13.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:00:13.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:12 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:00:13.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:00:13.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:00:13.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:00:13.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:12 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:00:14.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:13 smithi191 ceph-mon[33016]: pgmap v637: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:14.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:13 smithi053 ceph-mon[29182]: pgmap v637: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:14.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:13 smithi077 ceph-mon[32553]: pgmap v637: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:16.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:15 smithi191 ceph-mon[33016]: pgmap v638: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:16.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:15 smithi053 ceph-mon[29182]: pgmap v638: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:16.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:15 smithi077 ceph-mon[32553]: pgmap v638: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:18.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:17 smithi191 ceph-mon[33016]: pgmap v639: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:18.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:17 smithi053 ceph-mon[29182]: pgmap v639: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:18.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:17 smithi077 ceph-mon[32553]: pgmap v639: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:20.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:20 smithi191 ceph-mon[33016]: pgmap v640: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:20.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:20 smithi053 ceph-mon[29182]: pgmap v640: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:20.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:20 smithi077 ceph-mon[32553]: pgmap v640: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:22.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:22 smithi191 ceph-mon[33016]: pgmap v641: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:22.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:22 smithi053 ceph-mon[29182]: pgmap v641: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:22.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:22 smithi077 ceph-mon[32553]: pgmap v641: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:24.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:24 smithi191 ceph-mon[33016]: pgmap v642: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:24 smithi053 ceph-mon[29182]: pgmap v642: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:24.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:24 smithi077 ceph-mon[32553]: pgmap v642: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:26.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:26 smithi191 ceph-mon[33016]: pgmap v643: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:26.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:26 smithi053 ceph-mon[29182]: pgmap v643: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:26.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:26 smithi077 ceph-mon[32553]: pgmap v643: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:28.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:28 smithi191 ceph-mon[33016]: pgmap v644: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:28.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:28 smithi053 ceph-mon[29182]: pgmap v644: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:28.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:28 smithi077 ceph-mon[32553]: pgmap v644: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:30.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:30 smithi191 ceph-mon[33016]: pgmap v645: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:30.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:30 smithi053 ceph-mon[29182]: pgmap v645: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:30.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:30 smithi077 ceph-mon[32553]: pgmap v645: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:32 smithi191 ceph-mon[33016]: pgmap v646: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:32.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:32 smithi053 ceph-mon[29182]: pgmap v646: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:32.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:32 smithi077 ceph-mon[32553]: pgmap v646: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:34.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:34 smithi191 ceph-mon[33016]: pgmap v647: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:34.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:34 smithi053 ceph-mon[29182]: pgmap v647: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:34.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:34 smithi077 ceph-mon[32553]: pgmap v647: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:36.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:36 smithi191 ceph-mon[33016]: pgmap v648: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:36.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:36 smithi053 ceph-mon[29182]: pgmap v648: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:36.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:36 smithi077 ceph-mon[32553]: pgmap v648: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:38.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:38 smithi191 ceph-mon[33016]: pgmap v649: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:38.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:38 smithi053 ceph-mon[29182]: pgmap v649: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:38.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:38 smithi077 ceph-mon[32553]: pgmap v649: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:40.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:40 smithi191 ceph-mon[33016]: pgmap v650: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:40.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:40 smithi053 ceph-mon[29182]: pgmap v650: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:40.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:40 smithi077 ceph-mon[32553]: pgmap v650: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:42.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:42 smithi191 ceph-mon[33016]: pgmap v651: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:42.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:42 smithi053 ceph-mon[29182]: pgmap v651: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:42.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:42 smithi077 ceph-mon[32553]: pgmap v651: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:44 smithi053 ceph-mon[29182]: pgmap v652: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:44.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:44 smithi077 ceph-mon[32553]: pgmap v652: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:44.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:44 smithi191 ceph-mon[33016]: pgmap v652: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:46.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:46 smithi053 ceph-mon[29182]: pgmap v653: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:00:46.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:46 smithi077 ceph-mon[32553]: pgmap v653: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:00:46.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:46 smithi191 ceph-mon[33016]: pgmap v653: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:00:48.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:48 smithi053 ceph-mon[29182]: pgmap v654: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:48.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:48 smithi077 ceph-mon[32553]: pgmap v654: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:48.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:48 smithi191 ceph-mon[33016]: pgmap v654: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:50.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:50 smithi053 ceph-mon[29182]: pgmap v655: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:00:50.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:50 smithi077 ceph-mon[32553]: pgmap v655: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:00:50.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:50 smithi191 ceph-mon[33016]: pgmap v655: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:00:52.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:52 smithi191 ceph-mon[33016]: pgmap v656: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:00:52.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:52 smithi053 ceph-mon[29182]: pgmap v656: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:00:52.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:52 smithi077 ceph-mon[32553]: pgmap v656: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:00:54.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:54 smithi053 ceph-mon[29182]: pgmap v657: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:54.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:54 smithi077 ceph-mon[32553]: pgmap v657: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:54.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:54 smithi191 ceph-mon[33016]: pgmap v657: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:00:56.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:56 smithi053 ceph-mon[29182]: pgmap v658: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:56.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:56 smithi077 ceph-mon[32553]: pgmap v658: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:56.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:56 smithi191 ceph-mon[33016]: pgmap v658: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:00:58.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:00:58 smithi053 ceph-mon[29182]: pgmap v659: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:58.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:00:58 smithi077 ceph-mon[32553]: pgmap v659: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:58.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:00:58 smithi191 ceph-mon[33016]: pgmap v659: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:00.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:00 smithi053 ceph-mon[29182]: pgmap v660: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:00.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:00 smithi077 ceph-mon[32553]: pgmap v660: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:00.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:00 smithi191 ceph-mon[33016]: pgmap v660: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:02.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:02 smithi053 ceph-mon[29182]: pgmap v661: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:02.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:02 smithi077 ceph-mon[32553]: pgmap v661: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:02.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:02 smithi191 ceph-mon[33016]: pgmap v661: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:04.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:04 smithi053 ceph-mon[29182]: pgmap v662: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:04.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:04 smithi077 ceph-mon[32553]: pgmap v662: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:04.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:04 smithi191 ceph-mon[33016]: pgmap v662: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:06.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:06 smithi053 ceph-mon[29182]: pgmap v663: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:06.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:06 smithi077 ceph-mon[32553]: pgmap v663: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:06.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:06 smithi191 ceph-mon[33016]: pgmap v663: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:08.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:08 smithi053 ceph-mon[29182]: pgmap v664: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:08.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:08 smithi077 ceph-mon[32553]: pgmap v664: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:08.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:08 smithi191 ceph-mon[33016]: pgmap v664: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:10.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:10 smithi053 ceph-mon[29182]: pgmap v665: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:10.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:10 smithi077 ceph-mon[32553]: pgmap v665: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:10.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:10 smithi191 ceph-mon[33016]: pgmap v665: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:12.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:12 smithi053 ceph-mon[29182]: pgmap v666: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:12.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:12 smithi077 ceph-mon[32553]: pgmap v666: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:12.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:12 smithi191 ceph-mon[33016]: pgmap v666: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:13.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:13 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:01:13.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:13 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:01:13.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:13 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:01:14.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:14 smithi077 ceph-mon[32553]: pgmap v667: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:14.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:14 smithi053 ceph-mon[29182]: pgmap v667: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:14.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:14 smithi191 ceph-mon[33016]: pgmap v667: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:15.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: pgmap v668: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:01:15.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: pgmap v668: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:01:15.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: pgmap v668: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:01:15.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:01:15.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:01:18.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:17 smithi191 ceph-mon[33016]: pgmap v669: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:18.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:17 smithi053 ceph-mon[29182]: pgmap v669: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:18.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:17 smithi077 ceph-mon[32553]: pgmap v669: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:20.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:19 smithi191 ceph-mon[33016]: pgmap v670: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:20.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:19 smithi053 ceph-mon[29182]: pgmap v670: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:20.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:19 smithi077 ceph-mon[32553]: pgmap v670: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:22.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:21 smithi191 ceph-mon[33016]: pgmap v671: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:22.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:21 smithi053 ceph-mon[29182]: pgmap v671: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:22.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:21 smithi077 ceph-mon[32553]: pgmap v671: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:24.004 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:23 smithi077 ceph-mon[32553]: pgmap v672: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:24.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:23 smithi191 ceph-mon[33016]: pgmap v672: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:24.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:23 smithi053 ceph-mon[29182]: pgmap v672: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:26.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:25 smithi191 ceph-mon[33016]: pgmap v673: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:26.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:25 smithi053 ceph-mon[29182]: pgmap v673: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:26.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:25 smithi077 ceph-mon[32553]: pgmap v673: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:28.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:27 smithi191 ceph-mon[33016]: pgmap v674: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:28.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:27 smithi053 ceph-mon[29182]: pgmap v674: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:28.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:27 smithi077 ceph-mon[32553]: pgmap v674: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:30.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:29 smithi191 ceph-mon[33016]: pgmap v675: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:30.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:29 smithi053 ceph-mon[29182]: pgmap v675: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:30.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:29 smithi077 ceph-mon[32553]: pgmap v675: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:32.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:31 smithi191 ceph-mon[33016]: pgmap v676: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:32.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:31 smithi053 ceph-mon[29182]: pgmap v676: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:32.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:31 smithi077 ceph-mon[32553]: pgmap v676: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:34.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:33 smithi191 ceph-mon[33016]: pgmap v677: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:34.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:33 smithi053 ceph-mon[29182]: pgmap v677: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:34.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:33 smithi077 ceph-mon[32553]: pgmap v677: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:36.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:35 smithi191 ceph-mon[33016]: pgmap v678: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:36.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:35 smithi053 ceph-mon[29182]: pgmap v678: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:36.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:35 smithi077 ceph-mon[32553]: pgmap v678: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:38.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:37 smithi191 ceph-mon[33016]: pgmap v679: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:38.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:37 smithi053 ceph-mon[29182]: pgmap v679: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:38.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:37 smithi077 ceph-mon[32553]: pgmap v679: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:40.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:39 smithi191 ceph-mon[33016]: pgmap v680: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:40.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:39 smithi053 ceph-mon[29182]: pgmap v680: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:40.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:39 smithi077 ceph-mon[32553]: pgmap v680: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:42.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:41 smithi191 ceph-mon[33016]: pgmap v681: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:42.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:41 smithi053 ceph-mon[29182]: pgmap v681: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:42.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:41 smithi077 ceph-mon[32553]: pgmap v681: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:44.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:43 smithi191 ceph-mon[33016]: pgmap v682: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:44.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:43 smithi053 ceph-mon[29182]: pgmap v682: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:44.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:43 smithi077 ceph-mon[32553]: pgmap v682: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:46.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:45 smithi191 ceph-mon[33016]: pgmap v683: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:46.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:45 smithi053 ceph-mon[29182]: pgmap v683: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:46.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:45 smithi077 ceph-mon[32553]: pgmap v683: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:48.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:47 smithi191 ceph-mon[33016]: pgmap v684: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:48.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:47 smithi053 ceph-mon[29182]: pgmap v684: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:47 smithi077 ceph-mon[32553]: pgmap v684: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:50.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:49 smithi191 ceph-mon[33016]: pgmap v685: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:50.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:49 smithi053 ceph-mon[29182]: pgmap v685: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:50.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:49 smithi077 ceph-mon[32553]: pgmap v685: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:52.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:51 smithi191 ceph-mon[33016]: pgmap v686: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:52.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:51 smithi053 ceph-mon[29182]: pgmap v686: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:52.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:51 smithi077 ceph-mon[32553]: pgmap v686: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:54.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:53 smithi191 ceph-mon[33016]: pgmap v687: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:54.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:53 smithi053 ceph-mon[29182]: pgmap v687: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:54.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:53 smithi077 ceph-mon[32553]: pgmap v687: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:56.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:55 smithi191 ceph-mon[33016]: pgmap v688: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:56.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:55 smithi053 ceph-mon[29182]: pgmap v688: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:56.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:55 smithi077 ceph-mon[32553]: pgmap v688: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:01:58.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:57 smithi191 ceph-mon[33016]: pgmap v689: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:58.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:57 smithi053 ceph-mon[29182]: pgmap v689: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:58.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:57 smithi077 ceph-mon[32553]: pgmap v689: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:00.041 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:01:59 smithi191 ceph-mon[33016]: pgmap v690: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:00.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:01:59 smithi053 ceph-mon[29182]: pgmap v690: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:00.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:01:59 smithi077 ceph-mon[32553]: pgmap v690: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:02.041 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:01 smithi191 ceph-mon[33016]: pgmap v691: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:02.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:01 smithi053 ceph-mon[29182]: pgmap v691: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:02.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:01 smithi077 ceph-mon[32553]: pgmap v691: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:04.040 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:03 smithi191 ceph-mon[33016]: pgmap v692: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:04.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:03 smithi053 ceph-mon[29182]: pgmap v692: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:04.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:03 smithi077 ceph-mon[32553]: pgmap v692: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:06.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:05 smithi053 ceph-mon[29182]: pgmap v693: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:06.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:05 smithi077 ceph-mon[32553]: pgmap v693: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:06.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:05 smithi191 ceph-mon[33016]: pgmap v693: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:08.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:07 smithi053 ceph-mon[29182]: pgmap v694: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:08.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:07 smithi077 ceph-mon[32553]: pgmap v694: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:08.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:07 smithi191 ceph-mon[33016]: pgmap v694: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:10.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:09 smithi053 ceph-mon[29182]: pgmap v695: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:10.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:09 smithi077 ceph-mon[32553]: pgmap v695: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:10.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:09 smithi191 ceph-mon[33016]: pgmap v695: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:12.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:11 smithi053 ceph-mon[29182]: pgmap v696: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:12.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:11 smithi077 ceph-mon[32553]: pgmap v696: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:12.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:11 smithi191 ceph-mon[33016]: pgmap v696: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:14.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:13 smithi053 ceph-mon[29182]: pgmap v697: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:14.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:13 smithi077 ceph-mon[32553]: pgmap v697: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:14.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:13 smithi191 ceph-mon[33016]: pgmap v697: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:16.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:15 smithi053 ceph-mon[29182]: pgmap v698: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:16.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:15 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:02:16.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:15 smithi077 ceph-mon[32553]: pgmap v698: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:16.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:15 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:02:16.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:15 smithi191 ceph-mon[33016]: pgmap v698: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:16.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:15 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:02:17.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:02:17.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:02:17.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:02:17.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:02:17.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:02:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:02:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:02:18.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:17 smithi053 ceph-mon[29182]: pgmap v699: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:18.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:17 smithi077 ceph-mon[32553]: pgmap v699: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:18.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:17 smithi191 ceph-mon[33016]: pgmap v699: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:20.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:19 smithi053 ceph-mon[29182]: pgmap v700: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:20.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:19 smithi077 ceph-mon[32553]: pgmap v700: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:20.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:19 smithi191 ceph-mon[33016]: pgmap v700: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:22.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:21 smithi053 ceph-mon[29182]: pgmap v701: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:22.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:21 smithi077 ceph-mon[32553]: pgmap v701: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:22.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:21 smithi191 ceph-mon[33016]: pgmap v701: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:24.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:23 smithi053 ceph-mon[29182]: pgmap v702: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:24.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:23 smithi077 ceph-mon[32553]: pgmap v702: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:24.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:23 smithi191 ceph-mon[33016]: pgmap v702: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:26.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:25 smithi053 ceph-mon[29182]: pgmap v703: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:26.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:25 smithi077 ceph-mon[32553]: pgmap v703: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:26.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:25 smithi191 ceph-mon[33016]: pgmap v703: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:28.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:27 smithi053 ceph-mon[29182]: pgmap v704: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:02:28.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:27 smithi077 ceph-mon[32553]: pgmap v704: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:02:28.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:27 smithi191 ceph-mon[33016]: pgmap v704: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:02:30.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:29 smithi077 ceph-mon[32553]: pgmap v705: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:02:30.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:29 smithi191 ceph-mon[33016]: pgmap v705: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:02:30.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:29 smithi053 ceph-mon[29182]: pgmap v705: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:02:32.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:31 smithi077 ceph-mon[32553]: pgmap v706: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:31 smithi191 ceph-mon[33016]: pgmap v706: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:32.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:31 smithi053 ceph-mon[29182]: pgmap v706: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:34.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:33 smithi077 ceph-mon[32553]: pgmap v707: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:02:34.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:33 smithi191 ceph-mon[33016]: pgmap v707: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:02:34.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:33 smithi053 ceph-mon[29182]: pgmap v707: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:02:36.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:35 smithi077 ceph-mon[32553]: pgmap v708: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:36.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:35 smithi191 ceph-mon[33016]: pgmap v708: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:36.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:35 smithi053 ceph-mon[29182]: pgmap v708: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:02:38.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:37 smithi077 ceph-mon[32553]: pgmap v709: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:38.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:37 smithi191 ceph-mon[33016]: pgmap v709: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:38.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:37 smithi053 ceph-mon[29182]: pgmap v709: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:40.201 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:39 smithi053 ceph-mon[29182]: pgmap v710: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:40.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:39 smithi077 ceph-mon[32553]: pgmap v710: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:40.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:39 smithi191 ceph-mon[33016]: pgmap v710: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:42.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:02:42.018+0000 7f074cff9640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f07380060f0 con 0x7f0758102a80 2024-04-28T13:02:42.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:02:42.018+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f07380060f0 con 0x7f0758102a80 2024-04-28T13:02:42.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:41 smithi077 ceph-mon[32553]: pgmap v711: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:42.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:41 smithi191 ceph-mon[33016]: pgmap v711: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:42.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:41 smithi053 ceph-mon[29182]: pgmap v711: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:44.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:43 smithi077 ceph-mon[32553]: pgmap v712: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:44.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:43 smithi191 ceph-mon[33016]: pgmap v712: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:44.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:43 smithi053 ceph-mon[29182]: pgmap v712: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:46.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:45 smithi077 ceph-mon[32553]: pgmap v713: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:46.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:45 smithi191 ceph-mon[33016]: pgmap v713: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:46.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:45 smithi053 ceph-mon[29182]: pgmap v713: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:48.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:47 smithi077 ceph-mon[32553]: pgmap v714: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:48.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:47 smithi191 ceph-mon[33016]: pgmap v714: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:48.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:47 smithi053 ceph-mon[29182]: pgmap v714: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:50.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:49 smithi077 ceph-mon[32553]: pgmap v715: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:50.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:49 smithi191 ceph-mon[33016]: pgmap v715: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:50.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:49 smithi053 ceph-mon[29182]: pgmap v715: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:52.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:51 smithi077 ceph-mon[32553]: pgmap v716: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:52.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:51 smithi191 ceph-mon[33016]: pgmap v716: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:52.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:51 smithi053 ceph-mon[29182]: pgmap v716: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:54.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:53 smithi077 ceph-mon[32553]: pgmap v717: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:54.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:53 smithi191 ceph-mon[33016]: pgmap v717: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:54.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:53 smithi053 ceph-mon[29182]: pgmap v717: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:56.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:55 smithi077 ceph-mon[32553]: pgmap v718: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:56.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:55 smithi191 ceph-mon[33016]: pgmap v718: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:56.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:55 smithi053 ceph-mon[29182]: pgmap v718: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:02:58.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:57 smithi077 ceph-mon[32553]: pgmap v719: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:58.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:57 smithi191 ceph-mon[33016]: pgmap v719: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:58.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:57 smithi053 ceph-mon[29182]: pgmap v719: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:00.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:02:59 smithi077 ceph-mon[32553]: pgmap v720: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:00.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:02:59 smithi191 ceph-mon[33016]: pgmap v720: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:00.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:02:59 smithi053 ceph-mon[29182]: pgmap v720: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:02.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:01 smithi077 ceph-mon[32553]: pgmap v721: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:02.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:01 smithi191 ceph-mon[33016]: pgmap v721: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:02.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:01 smithi053 ceph-mon[29182]: pgmap v721: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:04.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:03 smithi077 ceph-mon[32553]: pgmap v722: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:04.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:03 smithi191 ceph-mon[33016]: pgmap v722: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:04.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:03 smithi053 ceph-mon[29182]: pgmap v722: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:06.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:05 smithi191 ceph-mon[33016]: pgmap v723: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:06.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:05 smithi053 ceph-mon[29182]: pgmap v723: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:06.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:05 smithi077 ceph-mon[32553]: pgmap v723: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:08.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:07 smithi191 ceph-mon[33016]: pgmap v724: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:08.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:07 smithi053 ceph-mon[29182]: pgmap v724: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:08.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:07 smithi077 ceph-mon[32553]: pgmap v724: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:10.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:09 smithi191 ceph-mon[33016]: pgmap v725: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:10.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:09 smithi053 ceph-mon[29182]: pgmap v725: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:10.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:09 smithi077 ceph-mon[32553]: pgmap v725: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:12.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:11 smithi191 ceph-mon[33016]: pgmap v726: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:12.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:11 smithi053 ceph-mon[29182]: pgmap v726: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:12.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:11 smithi077 ceph-mon[32553]: pgmap v726: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:14.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:13 smithi191 ceph-mon[33016]: pgmap v727: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:14.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:13 smithi053 ceph-mon[29182]: pgmap v727: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:14.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:13 smithi077 ceph-mon[32553]: pgmap v727: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:16.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:15 smithi191 ceph-mon[33016]: pgmap v728: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:16.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:15 smithi053 ceph-mon[29182]: pgmap v728: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:16.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:15 smithi077 ceph-mon[32553]: pgmap v728: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:17.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:16 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:03:17.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:16 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:03:17.451 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:16 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:03:18.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:18 smithi053 ceph-mon[29182]: pgmap v729: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:18.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:03:18.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:03:18.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:18 smithi077 ceph-mon[32553]: pgmap v729: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:18.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:03:18.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:03:18.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:18 smithi191 ceph-mon[33016]: pgmap v729: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:03:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:03:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:03:20.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:20 smithi053 ceph-mon[29182]: pgmap v730: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:20.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:20 smithi077 ceph-mon[32553]: pgmap v730: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:20.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:20 smithi191 ceph-mon[33016]: pgmap v730: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:22.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:22 smithi053 ceph-mon[29182]: pgmap v731: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:22.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:22 smithi077 ceph-mon[32553]: pgmap v731: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:22.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:22 smithi191 ceph-mon[33016]: pgmap v731: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:24.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:24 smithi191 ceph-mon[33016]: pgmap v732: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:24.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:24 smithi053 ceph-mon[29182]: pgmap v732: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:24.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:24 smithi077 ceph-mon[32553]: pgmap v732: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:26.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:26 smithi053 ceph-mon[29182]: pgmap v733: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:26.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:26 smithi077 ceph-mon[32553]: pgmap v733: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:26.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:26 smithi191 ceph-mon[33016]: pgmap v733: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:28.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:28 smithi053 ceph-mon[29182]: pgmap v734: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:28.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:28 smithi077 ceph-mon[32553]: pgmap v734: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:28.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:28 smithi191 ceph-mon[33016]: pgmap v734: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:30.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:30 smithi053 ceph-mon[29182]: pgmap v735: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:30.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:30 smithi077 ceph-mon[32553]: pgmap v735: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:30.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:30 smithi191 ceph-mon[33016]: pgmap v735: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:32.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:32 smithi053 ceph-mon[29182]: pgmap v736: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:32.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:32 smithi077 ceph-mon[32553]: pgmap v736: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:32.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:32 smithi191 ceph-mon[33016]: pgmap v736: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:34.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:34 smithi053 ceph-mon[29182]: pgmap v737: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:34.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:34 smithi077 ceph-mon[32553]: pgmap v737: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:34.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:34 smithi191 ceph-mon[33016]: pgmap v737: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:36.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:36 smithi053 ceph-mon[29182]: pgmap v738: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:36.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:36 smithi077 ceph-mon[32553]: pgmap v738: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:36.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:36 smithi191 ceph-mon[33016]: pgmap v738: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:38.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:38 smithi053 ceph-mon[29182]: pgmap v739: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:38.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:38 smithi077 ceph-mon[32553]: pgmap v739: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:38.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:38 smithi191 ceph-mon[33016]: pgmap v739: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:40.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:40 smithi053 ceph-mon[29182]: pgmap v740: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:40.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:40 smithi077 ceph-mon[32553]: pgmap v740: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:40.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:40 smithi191 ceph-mon[33016]: pgmap v740: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:42.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:42 smithi053 ceph-mon[29182]: pgmap v741: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:42.745 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:42 smithi077 ceph-mon[32553]: pgmap v741: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:42.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:42 smithi191 ceph-mon[33016]: pgmap v741: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:44.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:44 smithi053 ceph-mon[29182]: pgmap v742: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:44.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:44 smithi077 ceph-mon[32553]: pgmap v742: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:44.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:44 smithi191 ceph-mon[33016]: pgmap v742: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:46.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:46 smithi053 ceph-mon[29182]: pgmap v743: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:46.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:46 smithi077 ceph-mon[32553]: pgmap v743: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:46.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:46 smithi191 ceph-mon[33016]: pgmap v743: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:48.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:48 smithi053 ceph-mon[29182]: pgmap v744: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:48.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:48 smithi077 ceph-mon[32553]: pgmap v744: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:48.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:48 smithi191 ceph-mon[33016]: pgmap v744: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:50.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:50 smithi053 ceph-mon[29182]: pgmap v745: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:50.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:50 smithi077 ceph-mon[32553]: pgmap v745: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:50.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:50 smithi191 ceph-mon[33016]: pgmap v745: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:52.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:52 smithi053 ceph-mon[29182]: pgmap v746: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:52.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:52 smithi077 ceph-mon[32553]: pgmap v746: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:52.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:52 smithi191 ceph-mon[33016]: pgmap v746: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:54.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:54 smithi053 ceph-mon[29182]: pgmap v747: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:54.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:54 smithi077 ceph-mon[32553]: pgmap v747: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:54.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:54 smithi191 ceph-mon[33016]: pgmap v747: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:56.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:56 smithi053 ceph-mon[29182]: pgmap v748: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:56.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:56 smithi077 ceph-mon[32553]: pgmap v748: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:56.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:56 smithi191 ceph-mon[33016]: pgmap v748: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:03:58.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:03:58 smithi053 ceph-mon[29182]: pgmap v749: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:58.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:03:58 smithi077 ceph-mon[32553]: pgmap v749: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:58.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:03:58 smithi191 ceph-mon[33016]: pgmap v749: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:00.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:00 smithi053 ceph-mon[29182]: pgmap v750: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:00.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:00 smithi077 ceph-mon[32553]: pgmap v750: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:00.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:00 smithi191 ceph-mon[33016]: pgmap v750: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:02.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:02 smithi053 ceph-mon[29182]: pgmap v751: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:02.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:02 smithi077 ceph-mon[32553]: pgmap v751: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:02.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:02 smithi191 ceph-mon[33016]: pgmap v751: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:04.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:04 smithi053 ceph-mon[29182]: pgmap v752: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:04.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:04 smithi077 ceph-mon[32553]: pgmap v752: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:04.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:04 smithi191 ceph-mon[33016]: pgmap v752: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:06.650 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:06 smithi053 ceph-mon[29182]: pgmap v753: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:06.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:06 smithi077 ceph-mon[32553]: pgmap v753: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:06.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:06 smithi191 ceph-mon[33016]: pgmap v753: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:08.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:08 smithi053 ceph-mon[29182]: pgmap v754: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:08.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:08 smithi077 ceph-mon[32553]: pgmap v754: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:08.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:08 smithi191 ceph-mon[33016]: pgmap v754: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:10.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:10 smithi053 ceph-mon[29182]: pgmap v755: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:10.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:10 smithi077 ceph-mon[32553]: pgmap v755: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:10.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:10 smithi191 ceph-mon[33016]: pgmap v755: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:12.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:12 smithi053 ceph-mon[29182]: pgmap v756: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:12.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:12 smithi077 ceph-mon[32553]: pgmap v756: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:12.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:12 smithi191 ceph-mon[33016]: pgmap v756: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:14.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:14 smithi053 ceph-mon[29182]: pgmap v757: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:14.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:14 smithi077 ceph-mon[32553]: pgmap v757: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:14.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:14 smithi191 ceph-mon[33016]: pgmap v757: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:16.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:16 smithi077 ceph-mon[32553]: pgmap v758: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:16.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:16 smithi191 ceph-mon[33016]: pgmap v758: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:16.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:16 smithi053 ceph-mon[29182]: pgmap v758: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:18.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:18 smithi077 ceph-mon[32553]: pgmap v759: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:18.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:18 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:04:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:18 smithi191 ceph-mon[33016]: pgmap v759: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:18.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:18 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:04:18.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:18 smithi053 ceph-mon[29182]: pgmap v759: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:18.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:18 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:04:19.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:04:19.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:04:19.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:04:19.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:04:19.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:04:19.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:04:19.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:04:19.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:04:19.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:04:20.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:20 smithi077 ceph-mon[32553]: pgmap v760: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:20.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:20 smithi191 ceph-mon[33016]: pgmap v760: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:20.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:20 smithi053 ceph-mon[29182]: pgmap v760: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:22.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:22 smithi077 ceph-mon[32553]: pgmap v761: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:22.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:22 smithi191 ceph-mon[33016]: pgmap v761: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:22.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:22 smithi053 ceph-mon[29182]: pgmap v761: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:24.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:24 smithi077 ceph-mon[32553]: pgmap v762: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:24.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:24 smithi191 ceph-mon[33016]: pgmap v762: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:24.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:24 smithi053 ceph-mon[29182]: pgmap v762: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:26.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:26 smithi077 ceph-mon[32553]: pgmap v763: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:26.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:26 smithi191 ceph-mon[33016]: pgmap v763: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:26.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:26 smithi053 ceph-mon[29182]: pgmap v763: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:28.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:28 smithi077 ceph-mon[32553]: pgmap v764: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:28.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:28 smithi191 ceph-mon[33016]: pgmap v764: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:28.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:28 smithi053 ceph-mon[29182]: pgmap v764: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:30.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:30 smithi077 ceph-mon[32553]: pgmap v765: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:30.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:30 smithi191 ceph-mon[33016]: pgmap v765: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:30.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:30 smithi053 ceph-mon[29182]: pgmap v765: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:32.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:32 smithi077 ceph-mon[32553]: pgmap v766: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:32.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:32 smithi191 ceph-mon[33016]: pgmap v766: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:32.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:32 smithi053 ceph-mon[29182]: pgmap v766: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:34.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:34 smithi077 ceph-mon[32553]: pgmap v767: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:34.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:34 smithi191 ceph-mon[33016]: pgmap v767: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:34.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:34 smithi053 ceph-mon[29182]: pgmap v767: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:36.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:36 smithi077 ceph-mon[32553]: pgmap v768: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:36.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:36 smithi191 ceph-mon[33016]: pgmap v768: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:36.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:36 smithi053 ceph-mon[29182]: pgmap v768: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:38.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:38 smithi077 ceph-mon[32553]: pgmap v769: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:38.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:38 smithi191 ceph-mon[33016]: pgmap v769: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:38.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:38 smithi053 ceph-mon[29182]: pgmap v769: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:40.701 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:40 smithi053 ceph-mon[29182]: pgmap v770: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:40.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:40 smithi077 ceph-mon[32553]: pgmap v770: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:40.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:40 smithi191 ceph-mon[33016]: pgmap v770: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:42.701 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:42 smithi077 ceph-mon[32553]: pgmap v771: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:42.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:42 smithi191 ceph-mon[33016]: pgmap v771: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:42.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:42 smithi053 ceph-mon[29182]: pgmap v771: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:44.702 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:44 smithi077 ceph-mon[32553]: pgmap v772: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:44.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:44 smithi191 ceph-mon[33016]: pgmap v772: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:44.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:44 smithi053 ceph-mon[29182]: pgmap v772: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:46.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:46 smithi191 ceph-mon[33016]: pgmap v773: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:46.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:46 smithi053 ceph-mon[29182]: pgmap v773: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:46.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:46 smithi077 ceph-mon[32553]: pgmap v773: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:48.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:48 smithi191 ceph-mon[33016]: pgmap v774: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:48.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:48 smithi053 ceph-mon[29182]: pgmap v774: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:48.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:48 smithi077 ceph-mon[32553]: pgmap v774: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:50.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:50 smithi191 ceph-mon[33016]: pgmap v775: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:50.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:50 smithi053 ceph-mon[29182]: pgmap v775: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:50.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:50 smithi077 ceph-mon[32553]: pgmap v775: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:52.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:52 smithi191 ceph-mon[33016]: pgmap v776: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:52.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:52 smithi053 ceph-mon[29182]: pgmap v776: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:52.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:52 smithi077 ceph-mon[32553]: pgmap v776: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:54.754 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:54 smithi077 ceph-mon[32553]: pgmap v777: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:54.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:54 smithi191 ceph-mon[33016]: pgmap v777: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:54.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:54 smithi053 ceph-mon[29182]: pgmap v777: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:56.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:56 smithi191 ceph-mon[33016]: pgmap v778: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:56.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:56 smithi053 ceph-mon[29182]: pgmap v778: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:56.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:56 smithi077 ceph-mon[32553]: pgmap v778: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:04:58.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:04:58 smithi191 ceph-mon[33016]: pgmap v779: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:58.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:04:58 smithi053 ceph-mon[29182]: pgmap v779: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:58.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:04:58 smithi077 ceph-mon[32553]: pgmap v779: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:00.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:00 smithi191 ceph-mon[33016]: pgmap v780: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:00.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:00 smithi053 ceph-mon[29182]: pgmap v780: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:00.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:00 smithi077 ceph-mon[32553]: pgmap v780: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:02.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:02 smithi191 ceph-mon[33016]: pgmap v781: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:02.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:02 smithi053 ceph-mon[29182]: pgmap v781: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:02.951 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:02 smithi077 ceph-mon[32553]: pgmap v781: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:04.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:04 smithi191 ceph-mon[33016]: pgmap v782: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:04.899 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:04 smithi053 ceph-mon[29182]: pgmap v782: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:04.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:04 smithi077 ceph-mon[32553]: pgmap v782: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:05.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:05 smithi191 ceph-mon[33016]: pgmap v783: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:05.898 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:05 smithi053 ceph-mon[29182]: pgmap v783: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:05.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:05 smithi077 ceph-mon[32553]: pgmap v783: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:08.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:07 smithi053 ceph-mon[29182]: pgmap v784: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:08.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:07 smithi077 ceph-mon[32553]: pgmap v784: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:08.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:07 smithi191 ceph-mon[33016]: pgmap v784: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:10.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:09 smithi053 ceph-mon[29182]: pgmap v785: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:10.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:09 smithi077 ceph-mon[32553]: pgmap v785: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:10.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:09 smithi191 ceph-mon[33016]: pgmap v785: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:12.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:11 smithi053 ceph-mon[29182]: pgmap v786: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:12.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:11 smithi077 ceph-mon[32553]: pgmap v786: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:12.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:11 smithi191 ceph-mon[33016]: pgmap v786: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:14.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:13 smithi053 ceph-mon[29182]: pgmap v787: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:14.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:13 smithi077 ceph-mon[32553]: pgmap v787: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:14.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:13 smithi191 ceph-mon[33016]: pgmap v787: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:16.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:15 smithi053 ceph-mon[29182]: pgmap v788: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:16.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:15 smithi077 ceph-mon[32553]: pgmap v788: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:16.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:15 smithi191 ceph-mon[33016]: pgmap v788: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:18.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:17 smithi053 ceph-mon[29182]: pgmap v789: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:18.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:17 smithi077 ceph-mon[32553]: pgmap v789: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:18.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:17 smithi191 ceph-mon[33016]: pgmap v789: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:20.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:19 smithi053 ceph-mon[29182]: pgmap v790: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:20.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:05:20.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:05:20.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:05:20.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:19 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:05:20.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:19 smithi077 ceph-mon[32553]: pgmap v790: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:20.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:05:20.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:05:20.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:05:20.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:19 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:05:20.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:19 smithi191 ceph-mon[33016]: pgmap v790: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:20.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:05:20.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:05:20.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:05:20.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:19 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' 2024-04-28T13:05:22.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:21 smithi053 ceph-mon[29182]: pgmap v791: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:22.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:21 smithi077 ceph-mon[32553]: pgmap v791: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:22.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:21 smithi191 ceph-mon[33016]: pgmap v791: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:24.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:23 smithi191 ceph-mon[33016]: pgmap v792: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:24.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:23 smithi053 ceph-mon[29182]: pgmap v792: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:24.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:23 smithi077 ceph-mon[32553]: pgmap v792: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:26.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:25 smithi053 ceph-mon[29182]: pgmap v793: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:26.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:25 smithi077 ceph-mon[32553]: pgmap v793: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:26.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:25 smithi191 ceph-mon[33016]: pgmap v793: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:28.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:27 smithi053 ceph-mon[29182]: pgmap v794: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:28.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:27 smithi077 ceph-mon[32553]: pgmap v794: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:28.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:27 smithi191 ceph-mon[33016]: pgmap v794: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:30.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:29 smithi053 ceph-mon[29182]: pgmap v795: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:30.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:29 smithi077 ceph-mon[32553]: pgmap v795: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:30.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:29 smithi191 ceph-mon[33016]: pgmap v795: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:32.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:31 smithi053 ceph-mon[29182]: pgmap v796: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:32.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:31 smithi077 ceph-mon[32553]: pgmap v796: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:32.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:31 smithi191 ceph-mon[33016]: pgmap v796: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:34.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:33 smithi053 ceph-mon[29182]: pgmap v797: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:34.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:33 smithi077 ceph-mon[32553]: pgmap v797: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:34.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:33 smithi191 ceph-mon[33016]: pgmap v797: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:36.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:35 smithi053 ceph-mon[29182]: pgmap v798: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:36.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:35 smithi077 ceph-mon[32553]: pgmap v798: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:36.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:35 smithi191 ceph-mon[33016]: pgmap v798: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:05:38.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:37 smithi053 ceph-mon[29182]: pgmap v799: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:38.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:37 smithi077 ceph-mon[32553]: pgmap v799: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:38.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:37 smithi191 ceph-mon[33016]: pgmap v799: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:40.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:39 smithi053 ceph-mon[29182]: pgmap v800: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:40.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:39 smithi077 ceph-mon[32553]: pgmap v800: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:40.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:39 smithi191 ceph-mon[33016]: pgmap v800: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.934+0000 7f075cc68640 1 -- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400a3cf0 msgr2=0x7f07400a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900019910) for more than 900000000 us, fault. 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.934+0000 7f075cc68640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400a3cf0 0x7f07400a61b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0754000c00 tx=0x7f07540314a0 comp rx=0 tx=0).stop 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.934+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400a3cf0 msgr2=0x7f07400a61b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.934+0000 7f074dffb640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400a3cf0 0x7f07400a61b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.934+0000 7f074dffb640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400adda0 0x7f07400b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.934+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f0748015e80 con 0x7f07400adda0 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.935+0000 7f075cc68640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400adda0 0x7f07400b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:41.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:41.935+0000 7f075cc68640 1 --2- 172.21.15.53:0/1459642607 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f07400adda0 0x7f07400b0190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f07540314e0 tx=0x7f075403e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:42.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.045+0000 7f074cff9640 1 -- 172.21.15.53:0/1459642607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f0738006360 con 0x7f0758102a80 2024-04-28T13:05:42.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.045+0000 7f074dffb640 1 -- 172.21.15.53:0/1459642607 <== mon.0 v2:172.21.15.53:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f0738006360 con 0x7f0758102a80 2024-04-28T13:05:42.133 INFO:teuthology.orchestra.run.smithi053.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-28T13:05:42.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:41 smithi053 ceph-mon[29182]: pgmap v801: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-28T13:05:42.141 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-28T13:05:42.400 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=65dad90460f445ddb89f664dd60edbc9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-28T13:05:42.403 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-28T13:05:42.413 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-28T13:05:42.413 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T13:05:42.413 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:05:42.417 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:41 smithi191 ceph-mon[33016]: pgmap v801: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:42.418 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:41 smithi077 ceph-mon[32553]: pgmap v801: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:42.437 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-28T13:05:42.438 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:05:42.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:05:42.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 -- 172.21.15.53:0/354849442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04104550 msgr2=0x7f1b0410ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/354849442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04104550 0x7f1b0410ade0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1afc009ca0 tx=0x7f1afc02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 -- 172.21.15.53:0/354849442 shutdown_connections 2024-04-28T13:05:42.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/354849442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04104550 0x7f1b0410ade0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/354849442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b04103b90 0x7f1b04104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/354849442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b041029a0 0x7f1b04102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 -- 172.21.15.53:0/354849442 >> 172.21.15.53:0/354849442 conn(0x7f1b040fdf30 msgr2=0x7f1b04100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:42.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 -- 172.21.15.53:0/354849442 shutdown_connections 2024-04-28T13:05:42.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 -- 172.21.15.53:0/354849442 wait complete. 2024-04-28T13:05:42.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.653+0000 7f1b09b9d640 1 Processor -- start 2024-04-28T13:05:42.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b09b9d640 1 -- start start 2024-04-28T13:05:42.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b09b9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b041029a0 0x7f1b04198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b09b9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04103b90 0x7f1b041990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b09b9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b04104550 0x7f1b041a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b09b9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0410e4d0 con 0x7f1b04103b90 2024-04-28T13:05:42.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b09b9d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0410e350 con 0x7f1b041029a0 2024-04-28T13:05:42.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b09b9d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0410e650 con 0x7f1b04104550 2024-04-28T13:05:42.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04103b90 0x7f1b041990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04103b90 0x7f1b041990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55938/0 (socket says 172.21.15.53:55938) 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 -- 172.21.15.53:0/3690934526 learned_addr learned my addr 172.21.15.53:0/3690934526 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b03fff640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b04104550 0x7f1b041a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b037fe640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b041029a0 0x7f1b04198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 -- 172.21.15.53:0/3690934526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b04104550 msgr2=0x7f1b041a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b04104550 0x7f1b041a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 -- 172.21.15.53:0/3690934526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b041029a0 msgr2=0x7f1b04198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b041029a0 0x7f1b04198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 -- 172.21.15.53:0/3690934526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b041a07a0 con 0x7f1b04103b90 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b037fe640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b041029a0 0x7f1b04198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.654+0000 7f1b02ffd640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04103b90 0x7f1b041990c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1af800d950 tx=0x7f1af800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.655+0000 7f1b00ff9640 1 -- 172.21.15.53:0/3690934526 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8017030 con 0x7f1b04103b90 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.655+0000 7f1b00ff9640 1 -- 172.21.15.53:0/3690934526 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af8004020 con 0x7f1b04103b90 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.655+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b041a0a90 con 0x7f1b04103b90 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.655+0000 7f1b00ff9640 1 -- 172.21.15.53:0/3690934526 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af80047a0 con 0x7f1b04103b90 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.655+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b041a1030 con 0x7f1b04103b90 2024-04-28T13:05:42.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.656+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b04109d30 con 0x7f1b04103b90 2024-04-28T13:05:42.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.657+0000 7f1b00ff9640 1 -- 172.21.15.53:0/3690934526 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1af8009a30 con 0x7f1b04103b90 2024-04-28T13:05:42.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.657+0000 7f1b00ff9640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1aec0a3cc0 0x7f1aec0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.658+0000 7f1b037fe640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1aec0a3cc0 0x7f1aec0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.658+0000 7f1b00ff9640 1 -- 172.21.15.53:0/3690934526 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f1af80d0020 con 0x7f1b04103b90 2024-04-28T13:05:42.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.658+0000 7f1b037fe640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1aec0a3cc0 0x7f1aec0a6180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1b041039f0 tx=0x7f1af4005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:42.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.662+0000 7f1b00ff9640 1 -- 172.21.15.53:0/3690934526 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1af809c8a0 con 0x7f1b04103b90 2024-04-28T13:05:42.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.855+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b0402dcf0 con 0x7f1b04103b90 2024-04-28T13:05:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.856+0000 7f1b00ff9640 1 -- 172.21.15.53:0/3690934526 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7f1af809c1a0 con 0x7f1b04103b90 2024-04-28T13:05:42.856 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T13:05:42.857 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1aec0a3cc0 msgr2=0x7f1aec0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1aec0a3cc0 0x7f1aec0a6180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1b041039f0 tx=0x7f1af4005e70 comp rx=0 tx=0).stop 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04103b90 msgr2=0x7f1b041990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04103b90 0x7f1b041990c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1af800d950 tx=0x7f1af800de10 comp rx=0 tx=0).stop 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 shutdown_connections 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f1aec0a3cc0 0x7f1aec0a6180 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b04104550 0x7f1b041a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b04103b90 0x7f1b041990c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 --2- 172.21.15.53:0/3690934526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b041029a0 0x7f1b04198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.858+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 >> 172.21.15.53:0/3690934526 conn(0x7f1b040fdf30 msgr2=0x7f1b04106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:42.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.859+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 shutdown_connections 2024-04-28T13:05:42.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.859+0000 7f1b09b9d640 1 -- 172.21.15.53:0/3690934526 wait complete. 2024-04-28T13:05:42.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:44:00.459927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:42.284772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:42.832857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T13:05:42.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:05:42.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 -- 172.21.15.53:0/3050735476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94102980 msgr2=0x7f5c94102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/3050735476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94102980 0x7f5c94102da0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009900 tx=0x7f5c8802a410 comp rx=0 tx=0).stop 2024-04-28T13:05:42.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 -- 172.21.15.53:0/3050735476 shutdown_connections 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/3050735476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94104530 0x7f5c9410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/3050735476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c94103b70 0x7f5c94103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/3050735476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94102980 0x7f5c94102da0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 -- 172.21.15.53:0/3050735476 >> 172.21.15.53:0/3050735476 conn(0x7f5c940fdf10 msgr2=0x7f5c94100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 -- 172.21.15.53:0/3050735476 shutdown_connections 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.976+0000 7f5c9aa99640 1 -- 172.21.15.53:0/3050735476 wait complete. 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 Processor -- start 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 -- start start 2024-04-28T13:05:42.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94102980 0x7f5c94198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c94103b70 0x7f5c94199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94104530 0x7f5c941a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9410e480 con 0x7f5c94104530 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9410e300 con 0x7f5c94103b70 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c9aa99640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9410e600 con 0x7f5c94102980 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94104530 0x7f5c941a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94104530 0x7f5c941a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55964/0 (socket says 172.21.15.53:55964) 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 -- 172.21.15.53:0/2020177006 learned_addr learned my addr 172.21.15.53:0/2020177006 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:42.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c98835640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94102980 0x7f5c94198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c93fff640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c94103b70 0x7f5c94199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 -- 172.21.15.53:0/2020177006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94102980 msgr2=0x7f5c94198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94102980 0x7f5c94198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 -- 172.21.15.53:0/2020177006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c94103b70 msgr2=0x7f5c94199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c94103b70 0x7f5c94199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.977+0000 7f5c99036640 1 -- 172.21.15.53:0/2020177006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c941a0800 con 0x7f5c94104530 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.978+0000 7f5c99036640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94104530 0x7f5c941a0190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00eb10 tx=0x7f5c8c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:42.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.978+0000 7f5c91ffb640 1 -- 172.21.15.53:0/2020177006 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c004b20 con 0x7f5c94104530 2024-04-28T13:05:42.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.978+0000 7f5c91ffb640 1 -- 172.21.15.53:0/2020177006 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c8c004cc0 con 0x7f5c94104530 2024-04-28T13:05:42.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.978+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c941a0af0 con 0x7f5c94104530 2024-04-28T13:05:42.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.978+0000 7f5c91ffb640 1 -- 172.21.15.53:0/2020177006 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c010670 con 0x7f5c94104530 2024-04-28T13:05:42.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.978+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c941a1040 con 0x7f5c94104530 2024-04-28T13:05:42.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.979+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c94109e10 con 0x7f5c94104530 2024-04-28T13:05:42.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.980+0000 7f5c91ffb640 1 -- 172.21.15.53:0/2020177006 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5c8c005020 con 0x7f5c94104530 2024-04-28T13:05:42.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.980+0000 7f5c91ffb640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c840a3cc0 0x7f5c840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:42.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.981+0000 7f5c91ffb640 1 -- 172.21.15.53:0/2020177006 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f5c8c0d0c20 con 0x7f5c94104530 2024-04-28T13:05:42.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.981+0000 7f5c98835640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c840a3cc0 0x7f5c840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:42.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.981+0000 7f5c98835640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c840a3cc0 0x7f5c840a6180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5c88000c00 tx=0x7f5c880314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:42.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:42.985+0000 7f5c91ffb640 1 -- 172.21.15.53:0/2020177006 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c8c09d460 con 0x7f5c94104530 2024-04-28T13:05:43.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:42 smithi053 ceph-mon[29182]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:05:43.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.170+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5c9402dd20 con 0x7f5c94104530 2024-04-28T13:05:43.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.170+0000 7f5c91ffb640 1 -- 172.21.15.53:0/2020177006 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f5c8c09cd60 con 0x7f5c94104530 2024-04-28T13:05:43.170 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-28T13:05:43.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c840a3cc0 msgr2=0x7f5c840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c840a3cc0 0x7f5c840a6180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5c88000c00 tx=0x7f5c880314a0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94104530 msgr2=0x7f5c941a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94104530 0x7f5c941a0190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00eb10 tx=0x7f5c8c00c2d0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 shutdown_connections 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f5c840a3cc0 0x7f5c840a6180 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c94104530 0x7f5c941a0190 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c94103b70 0x7f5c94199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 --2- 172.21.15.53:0/2020177006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94102980 0x7f5c94198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 >> 172.21.15.53:0/2020177006 conn(0x7f5c940fdf10 msgr2=0x7f5c94106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:43.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 shutdown_connections 2024-04-28T13:05:43.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.172+0000 7f5c9aa99640 1 -- 172.21.15.53:0/2020177006 wait complete. 2024-04-28T13:05:43.183 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:05:43.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:42 smithi077 ceph-mon[32553]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:05:43.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 -- 172.21.15.53:0/2560785531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 msgr2=0x7ff180108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 --2- 172.21.15.53:0/2560785531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180108130 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff174009900 tx=0x7ff17402a410 comp rx=0 tx=0).stop 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 -- 172.21.15.53:0/2560785531 shutdown_connections 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 --2- 172.21.15.53:0/2560785531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff180108670 0x7ff18010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 --2- 172.21.15.53:0/2560785531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180108130 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 --2- 172.21.15.53:0/2560785531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff180103410 0x7ff180105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 -- 172.21.15.53:0/2560785531 >> 172.21.15.53:0/2560785531 conn(0x7ff1800fce20 msgr2=0x7ff1800ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 -- 172.21.15.53:0/2560785531 shutdown_connections 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 -- 172.21.15.53:0/2560785531 wait complete. 2024-04-28T13:05:43.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.289+0000 7ff186351640 1 Processor -- start 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff186351640 1 -- start start 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff186351640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff180103410 0x7ff180198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff186351640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff186351640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff180108670 0x7ff1801a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff186351640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff18010e4d0 con 0x7ff180105d40 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff186351640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff18010e350 con 0x7ff180103410 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff186351640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff18010e650 con 0x7ff180108670 2024-04-28T13:05:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55982/0 (socket says 172.21.15.53:55982) 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 -- 172.21.15.53:0/4202235271 learned_addr learned my addr 172.21.15.53:0/4202235271 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff1848ee640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff180108670 0x7ff1801a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17ffff640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff180103410 0x7ff180198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 -- 172.21.15.53:0/4202235271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff180108670 msgr2=0x7ff1801a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff180108670 0x7ff1801a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 -- 172.21.15.53:0/4202235271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff180103410 msgr2=0x7ff180198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff180103410 0x7ff180198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17f7fe640 1 -- 172.21.15.53:0/4202235271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1801a0830 con 0x7ff180105d40 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.290+0000 7ff17ffff640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff180103410 0x7ff180198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.291+0000 7ff17f7fe640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180199160 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff174006010 tx=0x7ff1740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.291+0000 7ff17d7fa640 1 -- 172.21.15.53:0/4202235271 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17403f030 con 0x7ff180105d40 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.291+0000 7ff17d7fa640 1 -- 172.21.15.53:0/4202235271 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1740318f0 con 0x7ff180105d40 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.291+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1801a0ac0 con 0x7ff180105d40 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.291+0000 7ff17d7fa640 1 -- 172.21.15.53:0/4202235271 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174035070 con 0x7ff180105d40 2024-04-28T13:05:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.291+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1801a1000 con 0x7ff180105d40 2024-04-28T13:05:43.296 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:42 smithi191 ceph-mon[33016]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:05:43.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.292+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1800fe910 con 0x7ff180105d40 2024-04-28T13:05:43.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.294+0000 7ff17d7fa640 1 -- 172.21.15.53:0/4202235271 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff174042030 con 0x7ff180105d40 2024-04-28T13:05:43.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.294+0000 7ff17d7fa640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff1680a3cc0 0x7ff1680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.294+0000 7ff17ffff640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff1680a3cc0 0x7ff1680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.294+0000 7ff17d7fa640 1 -- 172.21.15.53:0/4202235271 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7ff1740f0f00 con 0x7ff180105d40 2024-04-28T13:05:43.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.295+0000 7ff17ffff640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff1680a3cc0 0x7ff1680a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff18019a0f0 tx=0x7ff170005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:43.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.298+0000 7ff17d7fa640 1 -- 172.21.15.53:0/4202235271 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff1740bd780 con 0x7ff180105d40 2024-04-28T13:05:43.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.482+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff18002dd20 con 0x7ff180105d40 2024-04-28T13:05:43.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.483+0000 7ff17d7fa640 1 -- 172.21.15.53:0/4202235271 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7ff1740bd080 con 0x7ff180105d40 2024-04-28T13:05:43.483 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-04-28T13:05:43.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff1680a3cc0 msgr2=0x7ff1680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff1680a3cc0 0x7ff1680a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff18019a0f0 tx=0x7ff170005f90 comp rx=0 tx=0).stop 2024-04-28T13:05:43.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 msgr2=0x7ff180199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180199160 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff174006010 tx=0x7ff1740042f0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 shutdown_connections 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7ff1680a3cc0 0x7ff1680a6180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff180108670 0x7ff1801a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff180105d40 0x7ff180199160 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 --2- 172.21.15.53:0/4202235271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff180103410 0x7ff180198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 >> 172.21.15.53:0/4202235271 conn(0x7ff1800fce20 msgr2=0x7ff180103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 shutdown_connections 2024-04-28T13:05:43.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.485+0000 7ff186351640 1 -- 172.21.15.53:0/4202235271 wait complete. 2024-04-28T13:05:43.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:05:43.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.601+0000 7f527e124640 1 -- 172.21.15.53:0/2229442899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278103420 msgr2=0x7f5278109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.601+0000 7f527e124640 1 --2- 172.21.15.53:0/2229442899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278103420 0x7f5278109cb0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5270009ca0 tx=0x7f527002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 -- 172.21.15.53:0/2229442899 shutdown_connections 2024-04-28T13:05:43.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 --2- 172.21.15.53:0/2229442899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278103420 0x7f5278109cb0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 --2- 172.21.15.53:0/2229442899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5278102a60 0x7f5278102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 --2- 172.21.15.53:0/2229442899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278101870 0x7f5278101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 -- 172.21.15.53:0/2229442899 >> 172.21.15.53:0/2229442899 conn(0x7f52780fcde0 msgr2=0x7f52780ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:43.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 -- 172.21.15.53:0/2229442899 shutdown_connections 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 -- 172.21.15.53:0/2229442899 wait complete. 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 Processor -- start 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 -- start start 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278101870 0x7f5278198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.602+0000 7f527e124640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278102a60 0x7f52781990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f527e124640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5278103420 0x7f52781a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f527e124640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f527810e4c0 con 0x7f5278101870 2024-04-28T13:05:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f527e124640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f527810e340 con 0x7f5278103420 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f527e124640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f527810e640 con 0x7f5278102a60 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278101870 0x7f5278198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278101870 0x7f5278198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55998/0 (socket says 172.21.15.53:55998) 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 -- 172.21.15.53:0/1032278187 learned_addr learned my addr 172.21.15.53:0/1032278187 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f5277fff640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5278103420 0x7f52781a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f5276ffd640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278102a60 0x7f52781990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 -- 172.21.15.53:0/1032278187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278102a60 msgr2=0x7f52781990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278102a60 0x7f52781990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 -- 172.21.15.53:0/1032278187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5278103420 msgr2=0x7f52781a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5278103420 0x7f52781a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 -- 172.21.15.53:0/1032278187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52781a07c0 con 0x7f5278101870 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f52777fe640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278101870 0x7f5278198ba0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f526800e9c0 tx=0x7f526800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f5274ff9640 1 -- 172.21.15.53:0/1032278187 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526800cd80 con 0x7f5278101870 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f5274ff9640 1 -- 172.21.15.53:0/1032278187 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52680035f0 con 0x7f5278101870 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52781a0ab0 con 0x7f5278101870 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f5274ff9640 1 -- 172.21.15.53:0/1032278187 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5268010690 con 0x7f5278101870 2024-04-28T13:05:43.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.603+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52781a0fb0 con 0x7f5278101870 2024-04-28T13:05:43.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.604+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5278108c00 con 0x7f5278101870 2024-04-28T13:05:43.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.606+0000 7f5274ff9640 1 -- 172.21.15.53:0/1032278187 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5268004000 con 0x7f5278101870 2024-04-28T13:05:43.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.607+0000 7f5274ff9640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f52600a3c40 0x7f52600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.607+0000 7f5276ffd640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f52600a3c40 0x7f52600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.607+0000 7f5274ff9640 1 -- 172.21.15.53:0/1032278187 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f52680d0db0 con 0x7f5278101870 2024-04-28T13:05:43.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.607+0000 7f5276ffd640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f52600a3c40 0x7f52600a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f527819a0b0 tx=0x7f526c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:43.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.610+0000 7f5274ff9640 1 -- 172.21.15.53:0/1032278187 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f526809d5f0 con 0x7f5278101870 2024-04-28T13:05:43.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.797+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f527802dc60 con 0x7f5278101870 2024-04-28T13:05:43.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.797+0000 7f5274ff9640 1 -- 172.21.15.53:0/1032278187 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f526809cef0 con 0x7f5278101870 2024-04-28T13:05:43.797 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f52600a3c40 msgr2=0x7f52600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f52600a3c40 0x7f52600a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f527819a0b0 tx=0x7f526c005f90 comp rx=0 tx=0).stop 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278101870 msgr2=0x7f5278198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278101870 0x7f5278198ba0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f526800e9c0 tx=0x7f526800ee80 comp rx=0 tx=0).stop 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 shutdown_connections 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f52600a3c40 0x7f52600a6100 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5278103420 0x7f52781a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5278102a60 0x7f52781990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 --2- 172.21.15.53:0/1032278187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5278101870 0x7f5278198ba0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 >> 172.21.15.53:0/1032278187 conn(0x7f52780fcde0 msgr2=0x7f5278105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 shutdown_connections 2024-04-28T13:05:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.799+0000 7f527e124640 1 -- 172.21.15.53:0/1032278187 wait complete. 2024-04-28T13:05:43.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-28T13:05:43.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.914+0000 7f79e7afc640 1 -- 172.21.15.53:0/2637242220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0102a80 msgr2=0x7f79e0102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.914+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2637242220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0102a80 0x7f79e0102f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009900 tx=0x7f79d002a410 comp rx=0 tx=0).stop 2024-04-28T13:05:43.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- 172.21.15.53:0/2637242220 shutdown_connections 2024-04-28T13:05:43.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2637242220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e0103440 0x7f79e0109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2637242220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0102a80 0x7f79e0102f00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2637242220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e0101890 0x7f79e0101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- 172.21.15.53:0/2637242220 >> 172.21.15.53:0/2637242220 conn(0x7f79e00fce20 msgr2=0x7f79e00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:43.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- 172.21.15.53:0/2637242220 shutdown_connections 2024-04-28T13:05:43.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- 172.21.15.53:0/2637242220 wait complete. 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 Processor -- start 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- start start 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e0101890 0x7f79e0198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e0102a80 0x7f79e01991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0103440 0x7f79e01a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f79e010d3c0 con 0x7f79e0103440 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f79e010d240 con 0x7f79e0101890 2024-04-28T13:05:43.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e7afc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f79e010d540 con 0x7f79e0102a80 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e6099640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0103440 0x7f79e01a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.915+0000 7f79e6099640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0103440 0x7f79e01a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56006/0 (socket says 172.21.15.53:56006) 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e6099640 1 -- 172.21.15.53:0/2228628803 learned_addr learned my addr 172.21.15.53:0/2228628803 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e5097640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e0102a80 0x7f79e01991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e5898640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e0101890 0x7f79e0198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e6099640 1 -- 172.21.15.53:0/2228628803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e0102a80 msgr2=0x7f79e01991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e6099640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e0102a80 0x7f79e01991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e6099640 1 -- 172.21.15.53:0/2228628803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e0101890 msgr2=0x7f79e0198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:43.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e6099640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e0101890 0x7f79e0198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e6099640 1 -- 172.21.15.53:0/2228628803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e01a0850 con 0x7f79e0103440 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e5898640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e0101890 0x7f79e0198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e6099640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0103440 0x7f79e01a01b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f79d800eaa0 tx=0x7f79d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79d6ffd640 1 -- 172.21.15.53:0/2228628803 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d800cdb0 con 0x7f79e0103440 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79d6ffd640 1 -- 172.21.15.53:0/2228628803 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79d8005020 con 0x7f79e0103440 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e01a0b40 con 0x7f79e0103440 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79d6ffd640 1 -- 172.21.15.53:0/2228628803 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d80116d0 con 0x7f79e0103440 2024-04-28T13:05:43.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.916+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e01a1090 con 0x7f79e0103440 2024-04-28T13:05:43.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.917+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e01a1260 con 0x7f79e0103440 2024-04-28T13:05:43.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.918+0000 7f79d6ffd640 1 -- 172.21.15.53:0/2228628803 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f79d80051d0 con 0x7f79e0103440 2024-04-28T13:05:43.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.919+0000 7f79d6ffd640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f79cc0a3cc0 0x7f79cc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:43.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.919+0000 7f79d6ffd640 1 -- 172.21.15.53:0/2228628803 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7f79d80d0790 con 0x7f79e0103440 2024-04-28T13:05:43.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.919+0000 7f79e5898640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f79cc0a3cc0 0x7f79cc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:43.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.919+0000 7f79e5898640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f79cc0a3cc0 0x7f79cc0a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f79e019a110 tx=0x7f79dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:43.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:43.924+0000 7f79d6ffd640 1 -- 172.21.15.53:0/2228628803 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f79d8016030 con 0x7f79e0103440 2024-04-28T13:05:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.123+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f79e01a1430 con 0x7f79e0103440 2024-04-28T13:05:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.123+0000 7f79d6ffd640 1 -- 172.21.15.53:0/2228628803 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+591 (secure 0 0 0) 0x7f79d809d010 con 0x7f79e0103440 2024-04-28T13:05:44.123 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6800/2275002354 2024-04-29T12:41:34.367887+0000 2024-04-28T13:05:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/3169008100 2024-04-29T12:41:34.367887+0000 2024-04-28T13:05:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6801/3936438773 2024-04-29T12:41:06.674945+0000 2024-04-28T13:05:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/3416041324 2024-04-29T12:41:34.367887+0000 2024-04-28T13:05:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/1704277076 2024-04-29T12:41:34.367887+0000 2024-04-28T13:05:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/591838983 2024-04-29T12:41:06.674945+0000 2024-04-28T13:05:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6801/2275002354 2024-04-29T12:41:34.367887+0000 2024-04-28T13:05:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/2809885884 2024-04-29T12:41:06.674945+0000 2024-04-28T13:05:44.125 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/1598561968 2024-04-29T12:41:06.674945+0000 2024-04-28T13:05:44.125 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6800/3936438773 2024-04-29T12:41:06.674945+0000 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:listed 10 entries 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f79cc0a3cc0 msgr2=0x7f79cc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f79cc0a3cc0 0x7f79cc0a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f79e019a110 tx=0x7f79dc005f90 comp rx=0 tx=0).stop 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0103440 msgr2=0x7f79e01a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0103440 0x7f79e01a01b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f79d800eaa0 tx=0x7f79d800ef60 comp rx=0 tx=0).stop 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 shutdown_connections 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f79cc0a3cc0 0x7f79cc0a6180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e0103440 0x7f79e01a01b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79e0102a80 0x7f79e01991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 --2- 172.21.15.53:0/2228628803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e0101890 0x7f79e0198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 >> 172.21.15.53:0/2228628803 conn(0x7f79e00fce20 msgr2=0x7f79e00fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 shutdown_connections 2024-04-28T13:05:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.125+0000 7f79e7afc640 1 -- 172.21.15.53:0/2228628803 wait complete. 2024-04-28T13:05:44.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:43 smithi053 ceph-mon[29182]: pgmap v802: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:44.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3690934526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:05:44.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2020177006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4202235271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:43 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1032278187' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.137 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-28T13:05:44.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:05:44.137 DEBUG:teuthology.orchestra.run.smithi053:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-28T13:05:44.169 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-04-28T13:05:44.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:05:44.170 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-28T13:05:44.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:43 smithi077 ceph-mon[32553]: pgmap v802: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:44.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:43 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3690934526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:05:44.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:43 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2020177006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:43 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4202235271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:43 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1032278187' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:05:44.232 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-28T13:05:44.288 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-28T13:05:44.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:05:44.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-28T13:05:44.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:43 smithi191 ceph-mon[33016]: pgmap v802: 97 pgs: 97 active+clean; 635 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:44.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:43 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3690934526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:05:44.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:43 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2020177006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:43 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4202235271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:43 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1032278187' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:05:44.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:05:44.354 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-04-28T13:05:44.447 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-28T13:05:44.457 INFO:tasks.cephadm:Teardown begin 2024-04-28T13:05:44.457 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-28T13:05:44.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:05:44.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 -- 172.21.15.53:0/588341424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c104550 msgr2=0x7fce6c10ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 --2- 172.21.15.53:0/588341424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c104550 0x7fce6c10ade0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fce64009ca0 tx=0x7fce6402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 -- 172.21.15.53:0/588341424 shutdown_connections 2024-04-28T13:05:44.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 --2- 172.21.15.53:0/588341424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c104550 0x7fce6c10ade0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 --2- 172.21.15.53:0/588341424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6c103b90 0x7fce6c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 --2- 172.21.15.53:0/588341424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce6c1029a0 0x7fce6c102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 -- 172.21.15.53:0/588341424 >> 172.21.15.53:0/588341424 conn(0x7fce6c0fdf30 msgr2=0x7fce6c100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 -- 172.21.15.53:0/588341424 shutdown_connections 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 -- 172.21.15.53:0/588341424 wait complete. 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 Processor -- start 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.562+0000 7fce70978640 1 -- start start 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce70978640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6c1029a0 0x7fce6c198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce70978640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c103b90 0x7fce6c1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce70978640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce6c104550 0x7fce6c1a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce70978640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c10e4b0 con 0x7fce6c103b90 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce70978640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c10e330 con 0x7fce6c1029a0 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce70978640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c10e630 con 0x7fce6c104550 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c103b90 0x7fce6c1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c103b90 0x7fce6c1990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56020/0 (socket says 172.21.15.53:56020) 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 -- 172.21.15.53:0/1521648333 learned_addr learned my addr 172.21.15.53:0/1521648333 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce6a59c640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6c1029a0 0x7fce6c198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce6ad9d640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce6c104550 0x7fce6c1a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 -- 172.21.15.53:0/1521648333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce6c104550 msgr2=0x7fce6c1a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce6c104550 0x7fce6c1a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 -- 172.21.15.53:0/1521648333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6c1029a0 msgr2=0x7fce6c198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6c1029a0 0x7fce6c198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 -- 172.21.15.53:0/1521648333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6c1a07a0 con 0x7fce6c103b90 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce63fff640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c103b90 0x7fce6c1990c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e9f0 tx=0x7fce5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.563+0000 7fce637fe640 1 -- 172.21.15.53:0/1521648333 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5400cd80 con 0x7fce6c103b90 2024-04-28T13:05:44.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.564+0000 7fce637fe640 1 -- 172.21.15.53:0/1521648333 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce540108e0 con 0x7fce6c103b90 2024-04-28T13:05:44.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.564+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6c1a0a90 con 0x7fce6c103b90 2024-04-28T13:05:44.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.564+0000 7fce637fe640 1 -- 172.21.15.53:0/1521648333 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54018980 con 0x7fce6c103b90 2024-04-28T13:05:44.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.564+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce6c1a0fe0 con 0x7fce6c103b90 2024-04-28T13:05:44.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.565+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce6c109d40 con 0x7fce6c103b90 2024-04-28T13:05:44.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.566+0000 7fce637fe640 1 -- 172.21.15.53:0/1521648333 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fce54010430 con 0x7fce6c103b90 2024-04-28T13:05:44.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.566+0000 7fce637fe640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fce580a3c40 0x7fce580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.567+0000 7fce6a59c640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fce580a3c40 0x7fce580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.567+0000 7fce637fe640 1 -- 172.21.15.53:0/1521648333 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fce540d0d00 con 0x7fce6c103b90 2024-04-28T13:05:44.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.567+0000 7fce6a59c640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fce580a3c40 0x7fce580a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fce5c005fd0 tx=0x7fce5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:44.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.570+0000 7fce637fe640 1 -- 172.21.15.53:0/1521648333 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fce5409cf30 con 0x7fce6c103b90 2024-04-28T13:05:44.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.796+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce6c02dad0 con 0x7fce6c103b90 2024-04-28T13:05:44.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.796+0000 7fce637fe640 1 -- 172.21.15.53:0/1521648333 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7fce540a1230 con 0x7fce6c103b90 2024-04-28T13:05:44.796 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T13:05:44.797 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:50:40.825669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24478,24485]},"id":1}]} 2024-04-28T13:05:44.797 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.798+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fce580a3c40 msgr2=0x7fce580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.798+0000 7fce70978640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fce580a3c40 0x7fce580a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fce5c005fd0 tx=0x7fce5c00b410 comp rx=0 tx=0).stop 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.798+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c103b90 msgr2=0x7fce6c1990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.798+0000 7fce70978640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c103b90 0x7fce6c1990c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e9f0 tx=0x7fce5400eeb0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 shutdown_connections 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fce580a3c40 0x7fce580a6100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce6c104550 0x7fce6c1a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce6c103b90 0x7fce6c1990c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 --2- 172.21.15.53:0/1521648333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6c1029a0 0x7fce6c198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 >> 172.21.15.53:0/1521648333 conn(0x7fce6c0fdf30 msgr2=0x7fce6c106f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:44.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 shutdown_connections 2024-04-28T13:05:44.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.799+0000 7fce70978640 1 -- 172.21.15.53:0/1521648333 wait complete. 2024-04-28T13:05:44.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:05:44.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 -- 172.21.15.53:0/2374563538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 msgr2=0x7fcee4105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 --2- 172.21.15.53:0/2374563538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee4105fe0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fced8009900 tx=0x7fced802a410 comp rx=0 tx=0).stop 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 -- 172.21.15.53:0/2374563538 shutdown_connections 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 --2- 172.21.15.53:0/2374563538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcee4106520 0x7fcee41069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 --2- 172.21.15.53:0/2374563538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee4105fe0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 --2- 172.21.15.53:0/2374563538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcee4101200 0x7fcee4103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 -- 172.21.15.53:0/2374563538 >> 172.21.15.53:0/2374563538 conn(0x7fcee40fce20 msgr2=0x7fcee40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 -- 172.21.15.53:0/2374563538 shutdown_connections 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.913+0000 7fceea893640 1 -- 172.21.15.53:0/2374563538 wait complete. 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 Processor -- start 2024-04-28T13:05:44.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 -- start start 2024-04-28T13:05:44.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcee4101200 0x7fcee419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcee4106520 0x7fcee41a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcee4114ae0 con 0x7fcee4103b70 2024-04-28T13:05:44.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcee4114960 con 0x7fcee4101200 2024-04-28T13:05:44.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fceea893640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcee4114c60 con 0x7fcee4106520 2024-04-28T13:05:44.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee419f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56046/0 (socket says 172.21.15.53:56046) 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 -- 172.21.15.53:0/4109920295 learned_addr learned my addr 172.21.15.53:0/4109920295 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee8e30640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcee4106520 0x7fcee41a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee3fff640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcee4101200 0x7fcee419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 -- 172.21.15.53:0/4109920295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcee4106520 msgr2=0x7fcee41a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcee4106520 0x7fcee41a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 -- 172.21.15.53:0/4109920295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcee4101200 msgr2=0x7fcee419f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:44.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcee4101200 0x7fcee419f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee37fe640 1 -- 172.21.15.53:0/4109920295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee41a6df0 con 0x7fcee4103b70 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.914+0000 7fcee8e30640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcee4106520 0x7fcee41a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.915+0000 7fcee37fe640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee419f710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fced8005940 tx=0x7fced8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.915+0000 7fcee17fa640 1 -- 172.21.15.53:0/4109920295 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced803f030 con 0x7fcee4103b70 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.915+0000 7fcee17fa640 1 -- 172.21.15.53:0/4109920295 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fced8003cb0 con 0x7fcee4103b70 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.915+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee41a7080 con 0x7fcee4103b70 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.915+0000 7fcee17fa640 1 -- 172.21.15.53:0/4109920295 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8035070 con 0x7fcee4103b70 2024-04-28T13:05:44.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.915+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee41a7570 con 0x7fcee4103b70 2024-04-28T13:05:44.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.916+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee40fe910 con 0x7fcee4103b70 2024-04-28T13:05:44.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.917+0000 7fcee17fa640 1 -- 172.21.15.53:0/4109920295 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fced8042030 con 0x7fcee4103b70 2024-04-28T13:05:44.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.918+0000 7fcee17fa640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcecc0a3c40 0x7fcecc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:44.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.918+0000 7fcee3fff640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcecc0a3c40 0x7fcecc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:44.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.918+0000 7fcee17fa640 1 -- 172.21.15.53:0/4109920295 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fced80f11a0 con 0x7fcee4103b70 2024-04-28T13:05:44.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.918+0000 7fcee3fff640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcecc0a3c40 0x7fcecc0a6100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fced4005fd0 tx=0x7fced400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:44.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:44.921+0000 7fcee17fa640 1 -- 172.21.15.53:0/4109920295 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fced80bd9e0 con 0x7fcee4103b70 2024-04-28T13:05:45.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.144+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcee402dad0 con 0x7fcee4103b70 2024-04-28T13:05:45.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.145+0000 7fcee17fa640 1 -- 172.21.15.53:0/4109920295 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11873 (secure 0 0 0) 0x7fcee402dad0 con 0x7fcee4103b70 2024-04-28T13:05:45.145 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T13:05:45.145 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1145263488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1145263488},{"type":"v1","addr":"172.21.15.53:6835","nonce":1145263488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3609735401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":3609735401},{"type":"v1","addr":"172.21.15.53:6837","nonce":3609735401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/2476446586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":2476446586},{"type":"v1","addr":"172.21.15.53:6839","nonce":2476446586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/1978797303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":1978797303},{"type":"v1","addr":"172.21.15.53:6841","nonce":1978797303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":14646,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/2957868311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2957868311},{"type":"v1","addr":"172.21.15.77:6839","nonce":2957868311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/1697828385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1697828385},{"type":"v1","addr":"172.21.15.191:6833","nonce":1697828385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:43.873799+0000","modified":"2024-04-28T12:50:40.825669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24505,"mds_1":24485,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6833/3944921810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3944921810},{"type":"v1","addr":"172.21.15.77:6833","nonce":3944921810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6835/1670611904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1670611904},{"type":"v1","addr":"172.21.15.77:6835","nonce":1670611904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2819920278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2819920278},{"type":"v1","addr":"172.21.15.191:6835","nonce":2819920278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.191:6837/4132012864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":4132012864},{"type":"v1","addr":"172.21.15.191:6837","nonce":4132012864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.77:6837/2147382100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2147382100},{"type":"v1","addr":"172.21.15.77:6837","nonce":2147382100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/2399731075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":2399731075},{"type":"v1","addr":"172.21.15.191:6839","nonce":2399731075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24505,"qdb_cluster":[24505,24478,24485]},"id":1}]} 2024-04-28T13:05:45.146 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2024-04-28T13:05:45.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcecc0a3c40 msgr2=0x7fcecc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcecc0a3c40 0x7fcecc0a6100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fced4005fd0 tx=0x7fced400b390 comp rx=0 tx=0).stop 2024-04-28T13:05:45.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 msgr2=0x7fcee419f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee419f710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fced8005940 tx=0x7fced8004430 comp rx=0 tx=0).stop 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 shutdown_connections 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fcecc0a3c40 0x7fcecc0a6100 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcee4106520 0x7fcee41a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcee4103b70 0x7fcee419f710 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 --2- 172.21.15.53:0/4109920295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcee4101200 0x7fcee419f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 >> 172.21.15.53:0/4109920295 conn(0x7fcee40fce20 msgr2=0x7fcee4101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 shutdown_connections 2024-04-28T13:05:45.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.147+0000 7fceea893640 1 -- 172.21.15.53:0/4109920295 wait complete. 2024-04-28T13:05:45.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:44 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2228628803' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:05:45.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:44 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1521648333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:05:45.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:05:45.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:44 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2228628803' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:05:45.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:44 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1521648333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:05:45.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 -- 172.21.15.53:0/1092358228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44108660 msgr2=0x7fbd4410aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 --2- 172.21.15.53:0/1092358228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44108660 0x7fbd4410aab0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009ca0 tx=0x7fbd3c02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 -- 172.21.15.53:0/1092358228 shutdown_connections 2024-04-28T13:05:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 --2- 172.21.15.53:0/1092358228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44108660 0x7fbd4410aab0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 --2- 172.21.15.53:0/1092358228 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd44105d30 0x7fbd44108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 --2- 172.21.15.53:0/1092358228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd44103400 0x7fbd441057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 -- 172.21.15.53:0/1092358228 >> 172.21.15.53:0/1092358228 conn(0x7fbd440fce10 msgr2=0x7fbd440ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 -- 172.21.15.53:0/1092358228 shutdown_connections 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 -- 172.21.15.53:0/1092358228 wait complete. 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.260+0000 7fbd49b97640 1 Processor -- start 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd49b97640 1 -- start start 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd49b97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd44103400 0x7fbd44198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd49b97640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44105d30 0x7fbd441990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd49b97640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd44108660 0x7fbd441a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd49b97640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4410e4d0 con 0x7fbd44105d30 2024-04-28T13:05:45.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd49b97640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4410e350 con 0x7fbd44103400 2024-04-28T13:05:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd49b97640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4410e650 con 0x7fbd44108660 2024-04-28T13:05:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44105d30 0x7fbd441990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44105d30 0x7fbd441990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56058/0 (socket says 172.21.15.53:56058) 2024-04-28T13:05:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 -- 172.21.15.53:0/2811324372 learned_addr learned my addr 172.21.15.53:0/2811324372 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd437fe640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd44103400 0x7fbd44198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd43fff640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd44108660 0x7fbd441a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 -- 172.21.15.53:0/2811324372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd44108660 msgr2=0x7fbd441a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd44108660 0x7fbd441a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 -- 172.21.15.53:0/2811324372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd44103400 msgr2=0x7fbd44198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd44103400 0x7fbd44198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 -- 172.21.15.53:0/2811324372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd441a07c0 con 0x7fbd44105d30 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd437fe640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd44103400 0x7fbd44198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd43fff640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd44108660 0x7fbd441a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.261+0000 7fbd42ffd640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44105d30 0x7fbd441990e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c970 tx=0x7fbd3800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:45.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.262+0000 7fbd40ff9640 1 -- 172.21.15.53:0/2811324372 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38007bb0 con 0x7fbd44105d30 2024-04-28T13:05:45.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.262+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd441a0ab0 con 0x7fbd44105d30 2024-04-28T13:05:45.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.262+0000 7fbd40ff9640 1 -- 172.21.15.53:0/2811324372 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd38007d50 con 0x7fbd44105d30 2024-04-28T13:05:45.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.262+0000 7fbd40ff9640 1 -- 172.21.15.53:0/2811324372 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38018730 con 0x7fbd44105d30 2024-04-28T13:05:45.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.262+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd441a1000 con 0x7fbd44105d30 2024-04-28T13:05:45.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.263+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd440fe900 con 0x7fbd44105d30 2024-04-28T13:05:45.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.264+0000 7fbd40ff9640 1 -- 172.21.15.53:0/2811324372 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbd3800f990 con 0x7fbd44105d30 2024-04-28T13:05:45.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.264+0000 7fbd40ff9640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbd2c0a3c40 0x7fbd2c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.265+0000 7fbd437fe640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbd2c0a3c40 0x7fbd2c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.265+0000 7fbd40ff9640 1 -- 172.21.15.53:0/2811324372 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fbd380d0680 con 0x7fbd44105d30 2024-04-28T13:05:45.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.265+0000 7fbd437fe640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbd2c0a3c40 0x7fbd2c0a6100 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbd340058c0 tx=0x7fbd34005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:45.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.269+0000 7fbd40ff9640 1 -- 172.21.15.53:0/2811324372 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbd38014030 con 0x7fbd44105d30 2024-04-28T13:05:45.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:44 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2228628803' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:05:45.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:44 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1521648333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:05:45.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.458+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd4402dc80 con 0x7fbd44105d30 2024-04-28T13:05:45.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.460+0000 7fbd40ff9640 1 -- 172.21.15.53:0/2811324372 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22664 (secure 0 0 0) 0x7fbd3809cec0 con 0x7fbd44105d30 2024-04-28T13:05:45.460 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-28T13:05:45.460 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":88,"fsid":"10525726-055c-11ef-bc93-c7b262605968","created":"2024-04-28T12:40:39.137641+0000","modified":"2024-04-28T12:49:35.625293+0000","last_up_change":"2024-04-28T12:47:50.660713+0000","last_in_change":"2024-04-28T12:47:30.621621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:44:00.459927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:42.284772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:42.832857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fa9a9a7-7715-40fe-9f3f-4f494beb5dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6803","nonce":1076361705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6805","nonce":1076361705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6809","nonce":1076361705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1076361705},{"type":"v1","addr":"172.21.15.53:6807","nonce":1076361705}]},"public_addr":"172.21.15.53:6803/1076361705","cluster_addr":"172.21.15.53:6805/1076361705","heartbeat_back_addr":"172.21.15.53:6809/1076361705","heartbeat_front_addr":"172.21.15.53:6807/1076361705","state":["exists","up"]},{"osd":1,"uuid":"f15997c2-61f7-4970-b488-c38990780d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6801","nonce":2126226610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6803","nonce":2126226610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6807","nonce":2126226610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2126226610},{"type":"v1","addr":"172.21.15.77:6805","nonce":2126226610}]},"public_addr":"172.21.15.77:6801/2126226610","cluster_addr":"172.21.15.77:6803/2126226610","heartbeat_back_addr":"172.21.15.77:6807/2126226610","heartbeat_front_addr":"172.21.15.77:6805/2126226610","state":["exists","up"]},{"osd":2,"uuid":"129b78c4-4920-41b7-bf2e-b512e63ae231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6801","nonce":246351477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6803","nonce":246351477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6807","nonce":246351477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":246351477},{"type":"v1","addr":"172.21.15.191:6805","nonce":246351477}]},"public_addr":"172.21.15.191:6801/246351477","cluster_addr":"172.21.15.191:6803/246351477","heartbeat_back_addr":"172.21.15.191:6807/246351477","heartbeat_front_addr":"172.21.15.191:6805/246351477","state":["exists","up"]},{"osd":3,"uuid":"cbade960-1344-45b7-a7ed-a056a7500a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6811","nonce":3076698859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6813","nonce":3076698859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6817","nonce":3076698859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":3076698859},{"type":"v1","addr":"172.21.15.53:6815","nonce":3076698859}]},"public_addr":"172.21.15.53:6811/3076698859","cluster_addr":"172.21.15.53:6813/3076698859","heartbeat_back_addr":"172.21.15.53:6817/3076698859","heartbeat_front_addr":"172.21.15.53:6815/3076698859","state":["exists","up"]},{"osd":4,"uuid":"990ad0bc-63b6-4466-94ca-75ba51801c7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6809","nonce":3997878075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6811","nonce":3997878075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6815","nonce":3997878075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3997878075},{"type":"v1","addr":"172.21.15.77:6813","nonce":3997878075}]},"public_addr":"172.21.15.77:6809/3997878075","cluster_addr":"172.21.15.77:6811/3997878075","heartbeat_back_addr":"172.21.15.77:6815/3997878075","heartbeat_front_addr":"172.21.15.77:6813/3997878075","state":["exists","up"]},{"osd":5,"uuid":"3015dc0f-1a0e-4fc9-b38c-f779c500f3b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6809","nonce":2749306755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6811","nonce":2749306755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6815","nonce":2749306755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":2749306755},{"type":"v1","addr":"172.21.15.191:6813","nonce":2749306755}]},"public_addr":"172.21.15.191:6809/2749306755","cluster_addr":"172.21.15.191:6811/2749306755","heartbeat_back_addr":"172.21.15.191:6815/2749306755","heartbeat_front_addr":"172.21.15.191:6813/2749306755","state":["exists","up"]},{"osd":6,"uuid":"2591cbe0-d151-45b4-87c1-532c13a53037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6819","nonce":512872966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6821","nonce":512872966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6825","nonce":512872966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":512872966},{"type":"v1","addr":"172.21.15.53:6823","nonce":512872966}]},"public_addr":"172.21.15.53:6819/512872966","cluster_addr":"172.21.15.53:6821/512872966","heartbeat_back_addr":"172.21.15.53:6825/512872966","heartbeat_front_addr":"172.21.15.53:6823/512872966","state":["exists","up"]},{"osd":7,"uuid":"573bb4c3-4e03-48c7-ad54-90cc71ab0311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6817","nonce":1250050488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6819","nonce":1250050488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6823","nonce":1250050488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1250050488},{"type":"v1","addr":"172.21.15.77:6821","nonce":1250050488}]},"public_addr":"172.21.15.77:6817/1250050488","cluster_addr":"172.21.15.77:6819/1250050488","heartbeat_back_addr":"172.21.15.77:6823/1250050488","heartbeat_front_addr":"172.21.15.77:6821/1250050488","state":["exists","up"]},{"osd":8,"uuid":"44f526cb-dce8-4894-9cd7-21118f8d3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6817","nonce":1864030319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6819","nonce":1864030319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6823","nonce":1864030319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":1864030319},{"type":"v1","addr":"172.21.15.191:6821","nonce":1864030319}]},"public_addr":"172.21.15.191:6817/1864030319","cluster_addr":"172.21.15.191:6819/1864030319","heartbeat_back_addr":"172.21.15.191:6823/1864030319","heartbeat_front_addr":"172.21.15.191:6821/1864030319","state":["exists","up"]},{"osd":9,"uuid":"aec50a3e-2db8-4933-812d-3ac318ddccdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6827","nonce":3727947776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6829","nonce":3727947776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6833","nonce":3727947776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3727947776},{"type":"v1","addr":"172.21.15.53:6831","nonce":3727947776}]},"public_addr":"172.21.15.53:6827/3727947776","cluster_addr":"172.21.15.53:6829/3727947776","heartbeat_back_addr":"172.21.15.53:6833/3727947776","heartbeat_front_addr":"172.21.15.53:6831/3727947776","state":["exists","up"]},{"osd":10,"uuid":"e1817077-d7a5-4eeb-8e2a-b948147d22a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6825","nonce":2556585572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6827","nonce":2556585572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6831","nonce":2556585572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2556585572},{"type":"v1","addr":"172.21.15.77:6829","nonce":2556585572}]},"public_addr":"172.21.15.77:6825/2556585572","cluster_addr":"172.21.15.77:6827/2556585572","heartbeat_back_addr":"172.21.15.77:6831/2556585572","heartbeat_front_addr":"172.21.15.77:6829/2556585572","state":["exists","up"]},{"osd":11,"uuid":"4c0e510f-d109-48e1-844c-1e2146a8a412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6825","nonce":1302539827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6827","nonce":1302539827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6831","nonce":1302539827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1302539827},{"type":"v1","addr":"172.21.15.191:6829","nonce":1302539827}]},"public_addr":"172.21.15.191:6825/1302539827","cluster_addr":"172.21.15.191:6827/1302539827","heartbeat_back_addr":"172.21.15.191:6831/1302539827","heartbeat_front_addr":"172.21.15.191:6829/1302539827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:09.029622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:32.623351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:56.809500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:22.045399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:47.605204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.537382+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:38.521224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:04.092971+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:29.441670+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:56.244479+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:22.578133+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:49.059632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6800/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/3169008100":"2024-04-29T12:41:34.367887+0000","172.21.15.53:6801/3936438773":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/3416041324":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/1704277076":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/591838983":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6801/2275002354":"2024-04-29T12:41:34.367887+0000","172.21.15.53:0/2809885884":"2024-04-29T12:41:06.674945+0000","172.21.15.53:0/1598561968":"2024-04-29T12:41:06.674945+0000","172.21.15.53:6800/3936438773":"2024-04-29T12:41:06.674945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:05:45.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbd2c0a3c40 msgr2=0x7fbd2c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbd2c0a3c40 0x7fbd2c0a6100 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbd340058c0 tx=0x7fbd34005850 comp rx=0 tx=0).stop 2024-04-28T13:05:45.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44105d30 msgr2=0x7fbd441990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44105d30 0x7fbd441990e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c970 tx=0x7fbd3800ce30 comp rx=0 tx=0).stop 2024-04-28T13:05:45.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 shutdown_connections 2024-04-28T13:05:45.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fbd2c0a3c40 0x7fbd2c0a6100 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd44108660 0x7fbd441a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd44105d30 0x7fbd441990e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 --2- 172.21.15.53:0/2811324372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd44103400 0x7fbd44198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 >> 172.21.15.53:0/2811324372 conn(0x7fbd440fce10 msgr2=0x7fbd44103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:45.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 shutdown_connections 2024-04-28T13:05:45.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.462+0000 7fbd49b97640 1 -- 172.21.15.53:0/2811324372 wait complete. 2024-04-28T13:05:45.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-28T13:05:45.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 -- 172.21.15.53:0/1901907574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88102990 msgr2=0x7fdc88102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/1901907574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88102990 0x7fdc88102db0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009900 tx=0x7fdc8402a410 comp rx=0 tx=0).stop 2024-04-28T13:05:45.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 -- 172.21.15.53:0/1901907574 shutdown_connections 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/1901907574 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc88104540 0x7fdc8810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/1901907574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc88103b80 0x7fdc88104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/1901907574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88102990 0x7fdc88102db0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 -- 172.21.15.53:0/1901907574 >> 172.21.15.53:0/1901907574 conn(0x7fdc880fdf00 msgr2=0x7fdc88100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 -- 172.21.15.53:0/1901907574 shutdown_connections 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 -- 172.21.15.53:0/1901907574 wait complete. 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.576+0000 7fdc8fc93640 1 Processor -- start 2024-04-28T13:05:45.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8fc93640 1 -- start start 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8fc93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc88102990 0x7fdc88198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8fc93640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc88103b80 0x7fdc88199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8fc93640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88104540 0x7fdc881a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8fc93640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8810e490 con 0x7fdc88104540 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8fc93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8810e310 con 0x7fdc88102990 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8fc93640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8810e610 con 0x7fdc88103b80 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88104540 0x7fdc881a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88104540 0x7fdc881a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56072/0 (socket says 172.21.15.53:56072) 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 -- 172.21.15.53:0/4087443434 learned_addr learned my addr 172.21.15.53:0/4087443434 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8d22e640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc88103b80 0x7fdc88199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8da2f640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc88102990 0x7fdc88198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 -- 172.21.15.53:0/4087443434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc88103b80 msgr2=0x7fdc88199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc88103b80 0x7fdc88199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 -- 172.21.15.53:0/4087443434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc88102990 msgr2=0x7fdc88198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc88102990 0x7fdc88198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 -- 172.21.15.53:0/4087443434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc881a07f0 con 0x7fdc88104540 2024-04-28T13:05:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc8e230640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88104540 0x7fdc881a0180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000eaa0 tx=0x7fdc8000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:45.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.577+0000 7fdc7effd640 1 -- 172.21.15.53:0/4087443434 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000cdb0 con 0x7fdc88104540 2024-04-28T13:05:45.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.578+0000 7fdc7effd640 1 -- 172.21.15.53:0/4087443434 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc80005020 con 0x7fdc88104540 2024-04-28T13:05:45.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.578+0000 7fdc7effd640 1 -- 172.21.15.53:0/4087443434 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc800116d0 con 0x7fdc88104540 2024-04-28T13:05:45.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.578+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc881a0ae0 con 0x7fdc88104540 2024-04-28T13:05:45.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.578+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc881a1030 con 0x7fdc88104540 2024-04-28T13:05:45.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.578+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc88109d20 con 0x7fdc88104540 2024-04-28T13:05:45.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.580+0000 7fdc7effd640 1 -- 172.21.15.53:0/4087443434 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdc800051d0 con 0x7fdc88104540 2024-04-28T13:05:45.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.581+0000 7fdc7effd640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdc740a3cc0 0x7fdc740a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.581+0000 7fdc8da2f640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdc740a3cc0 0x7fdc740a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.581+0000 7fdc7effd640 1 -- 172.21.15.53:0/4087443434 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8240+0+0 (secure 0 0 0) 0x7fdc800d0a20 con 0x7fdc88104540 2024-04-28T13:05:45.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.581+0000 7fdc8da2f640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdc740a3cc0 0x7fdc740a6180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009230 tx=0x7fdc840314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:45.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.584+0000 7fdc7effd640 1 -- 172.21.15.53:0/4087443434 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdc80016030 con 0x7fdc88104540 2024-04-28T13:05:45.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.810+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fdc8802dd20 con 0x7fdc88104540 2024-04-28T13:05:45.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.813+0000 7fdc7effd640 1 -- 172.21.15.53:0/4087443434 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7fdc8009ccd0 con 0x7fdc88104540 2024-04-28T13:05:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-28T13:05:45.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdc740a3cc0 msgr2=0x7fdc740a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdc740a3cc0 0x7fdc740a6180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009230 tx=0x7fdc840314a0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88104540 msgr2=0x7fdc881a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88104540 0x7fdc881a0180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000eaa0 tx=0x7fdc8000ef60 comp rx=0 tx=0).stop 2024-04-28T13:05:45.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 shutdown_connections 2024-04-28T13:05:45.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7fdc740a3cc0 0x7fdc740a6180 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc88104540 0x7fdc881a0180 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc88103b80 0x7fdc88199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 --2- 172.21.15.53:0/4087443434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc88102990 0x7fdc88198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 >> 172.21.15.53:0/4087443434 conn(0x7fdc880fdf00 msgr2=0x7fdc88106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:45.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 shutdown_connections 2024-04-28T13:05:45.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.816+0000 7fdc8fc93640 1 -- 172.21.15.53:0/4087443434 wait complete. 2024-04-28T13:05:45.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-28T13:05:45.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.934+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1093603945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8103440 msgr2=0x7f9cb8109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.934+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1093603945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8103440 0x7f9cb8109cd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009ca0 tx=0x7f9cb002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.934+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1093603945 shutdown_connections 2024-04-28T13:05:45.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.934+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1093603945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8103440 0x7f9cb8109cd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.934+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1093603945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cb8102a80 0x7f9cb8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.934+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1093603945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8101890 0x7f9cb8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.934+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1093603945 >> 172.21.15.53:0/1093603945 conn(0x7f9cb80fce20 msgr2=0x7f9cb80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1093603945 shutdown_connections 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1093603945 wait complete. 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 Processor -- start 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 -- start start 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8101890 0x7f9cb8198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8102a80 0x7f9cb81990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cb8103440 0x7f9cb81a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb810d3c0 con 0x7f9cb8102a80 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb810d240 con 0x7f9cb8101890 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbf0fc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb810d540 con 0x7f9cb8103440 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cb7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8102a80 0x7f9cb81990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cb7fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8102a80 0x7f9cb81990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56084/0 (socket says 172.21.15.53:56084) 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cb7fff640 1 -- 172.21.15.53:0/1940951253 learned_addr learned my addr 172.21.15.53:0/1940951253 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:45.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.935+0000 7f9cbce98640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8101890 0x7f9cb8198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cbd699640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cb8103440 0x7f9cb81a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb7fff640 1 -- 172.21.15.53:0/1940951253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cb8103440 msgr2=0x7f9cb81a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb7fff640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cb8103440 0x7f9cb81a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb7fff640 1 -- 172.21.15.53:0/1940951253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8101890 msgr2=0x7f9cb8198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb7fff640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8101890 0x7f9cb8198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb7fff640 1 -- 172.21.15.53:0/1940951253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb81a07a0 con 0x7f9cb8102a80 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cbce98640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8101890 0x7f9cb8198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:05:45.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb7fff640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8102a80 0x7f9cb81990c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00e9f0 tx=0x7f9cac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:45.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb5ffb640 1 -- 172.21.15.53:0/1940951253 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00cd80 con 0x7f9cb8102a80 2024-04-28T13:05:45.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb5ffb640 1 -- 172.21.15.53:0/1940951253 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cac0054d0 con 0x7f9cb8102a80 2024-04-28T13:05:45.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb81a0a90 con 0x7f9cb8102a80 2024-04-28T13:05:45.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cb5ffb640 1 -- 172.21.15.53:0/1940951253 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac010690 con 0x7f9cb8102a80 2024-04-28T13:05:45.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.936+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb81a0fe0 con 0x7f9cb8102a80 2024-04-28T13:05:45.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.937+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cb81a1270 con 0x7f9cb8102a80 2024-04-28T13:05:45.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.938+0000 7f9cb5ffb640 1 -- 172.21.15.53:0/1940951253 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9cac004e90 con 0x7f9cb8102a80 2024-04-28T13:05:45.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.939+0000 7f9cb5ffb640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ca00a3cc0 0x7f9ca00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:45.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.939+0000 7f9cb5ffb640 1 -- 172.21.15.53:0/1940951253 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8498+0+0 (secure 0 0 0) 0x7f9cac0d0a90 con 0x7f9cb8102a80 2024-04-28T13:05:45.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.939+0000 7f9cbce98640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ca00a3cc0 0x7f9ca00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:45.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.939+0000 7f9cbce98640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ca00a3cc0 0x7f9ca00a6180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9cb81028e0 tx=0x7f9ca8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:45.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:45.944+0000 7f9cb5ffb640 1 -- 172.21.15.53:0/1940951253 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9cac09d1a0 con 0x7f9cb8102a80 2024-04-28T13:05:46.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a[29178]: 2024-04-28T13:05:45.811+0000 7fe25c354700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:05:46.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: pgmap v803: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-28T13:05:46.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4109920295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:05:46.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/2811324372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:05:46.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4087443434' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:05:46.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:05:46.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:05:46.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/4087443434' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:05:46.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T13:05:46.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:45 smithi053 ceph-mon[29182]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:05:46.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.168+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9cb81a14e0 con 0x7f9cb8102a80 2024-04-28T13:05:46.202 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e[62416]: ignoring --setuser ceph since I am not root 2024-04-28T13:05:46.202 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e[62416]: ignoring --setgroup ceph since I am not root 2024-04-28T13:05:46.202 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mds[62420]: -- 172.21.15.77:0/1361272901 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b6fdac1a40 con 0x55b6fda94800 2024-04-28T13:05:46.202 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e[62416]: starting mds.e at 2024-04-28T13:05:46.203 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h[62723]: ignoring --setuser ceph since I am not root 2024-04-28T13:05:46.203 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h[62723]: ignoring --setgroup ceph since I am not root 2024-04-28T13:05:46.203 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mds[62727]: -- 172.21.15.77:0/1482468118 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5574d2e9fa40 con 0x5574d2e72800 2024-04-28T13:05:46.203 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h[62723]: starting mds.h at 2024-04-28T13:05:46.204 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k[62100]: ignoring --setuser ceph since I am not root 2024-04-28T13:05:46.204 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k[62100]: ignoring --setgroup ceph since I am not root 2024-04-28T13:05:46.204 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mds[62104]: -- 172.21.15.77:0/2549353332 <== mon.2 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab65141a40 con 0x55ab65114800 2024-04-28T13:05:46.204 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k[62100]: starting mds.k at 2024-04-28T13:05:46.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: pgmap v803: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-28T13:05:46.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4109920295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:05:46.205 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/2811324372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:05:46.205 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4087443434' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:05:46.205 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:05:46.205 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:05:46.205 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/4087443434' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:05:46.205 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T13:05:46.205 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:45 smithi077 ceph-mon[32553]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:05:46.291 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f[63570]: ignoring --setuser ceph since I am not root 2024-04-28T13:05:46.291 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f[63570]: ignoring --setgroup ceph since I am not root 2024-04-28T13:05:46.291 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mds[63574]: -- 172.21.15.191:0/1579123716 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563dce2c7a40 con 0x563dce29ac00 2024-04-28T13:05:46.291 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f[63570]: starting mds.f at 2024-04-28T13:05:46.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: pgmap v803: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-28T13:05:46.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4109920295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:05:46.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/2811324372' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:05:46.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4087443434' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:05:46.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:05:46.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:05:46.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/4087443434' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:05:46.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T13:05:46.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mon[33016]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:05:46.293 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c[63883]: ignoring --setuser ceph since I am not root 2024-04-28T13:05:46.294 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c[63883]: ignoring --setgroup ceph since I am not root 2024-04-28T13:05:46.294 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mds[63887]: -- 172.21.15.191:0/3343343533 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a403eba40 con 0x559a403be800 2024-04-28T13:05:46.294 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c[63883]: starting mds.c at 2024-04-28T13:05:46.294 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: ignoring --setuser ceph since I am not root 2024-04-28T13:05:46.294 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: ignoring --setgroup ceph since I am not root 2024-04-28T13:05:46.294 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-mds[64197]: -- 172.21.15.191:0/3768885261 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d32cd17a40 con 0x55d32cceac00 2024-04-28T13:05:46.295 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:05:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: starting mds.i at 2024-04-28T13:05:46.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.860+0000 7f9cb5ffb640 1 -- 172.21.15.53:0/1940951253 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f9cac09caa0 con 0x7f9cb8102a80 2024-04-28T13:05:46.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ca00a3cc0 msgr2=0x7f9ca00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:46.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ca00a3cc0 0x7f9ca00a6180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9cb81028e0 tx=0x7f9ca8005f90 comp rx=0 tx=0).stop 2024-04-28T13:05:46.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8102a80 msgr2=0x7f9cb81990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:46.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8102a80 0x7f9cb81990c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00e9f0 tx=0x7f9cac00edf0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 shutdown_connections 2024-04-28T13:05:46.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9ca00a3cc0 0x7f9ca00a6180 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cb8103440 0x7f9cb81a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cb8102a80 0x7f9cb81990c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.863+0000 7f9cbf0fc640 1 --2- 172.21.15.53:0/1940951253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8101890 0x7f9cb8198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.864+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 >> 172.21.15.53:0/1940951253 conn(0x7f9cb80fce20 msgr2=0x7f9cb80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:46.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.864+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 shutdown_connections 2024-04-28T13:05:46.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.864+0000 7f9cbf0fc640 1 -- 172.21.15.53:0/1940951253 wait complete. 2024-04-28T13:05:46.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-28T13:05:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- 172.21.15.53:0/1450210206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320102a60 msgr2=0x7f0320102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 --2- 172.21.15.53:0/1450210206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320102a60 0x7f0320102ee0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0310009900 tx=0x7f031002a410 comp rx=0 tx=0).stop 2024-04-28T13:05:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- 172.21.15.53:0/1450210206 shutdown_connections 2024-04-28T13:05:46.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 --2- 172.21.15.53:0/1450210206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0320103420 0x7f0320109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 --2- 172.21.15.53:0/1450210206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320102a60 0x7f0320102ee0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 --2- 172.21.15.53:0/1450210206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0320101870 0x7f0320101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- 172.21.15.53:0/1450210206 >> 172.21.15.53:0/1450210206 conn(0x7f03200fcde0 msgr2=0x7f03200ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- 172.21.15.53:0/1450210206 shutdown_connections 2024-04-28T13:05:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- 172.21.15.53:0/1450210206 wait complete. 2024-04-28T13:05:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 Processor -- start 2024-04-28T13:05:46.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- start start 2024-04-28T13:05:46.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320101870 0x7f032006a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:46.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0320102a60 0x7f032019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:46.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0320103420 0x7f032019fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:46.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f032010d3f0 con 0x7f0320101870 2024-04-28T13:05:46.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f032010d270 con 0x7f0320102a60 2024-04-28T13:05:46.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.994+0000 7f0326be7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f032010d570 con 0x7f0320103420 2024-04-28T13:05:46.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320101870 0x7f032006a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:46.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320101870 0x7f032006a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50954/0 (socket says 172.21.15.53:50954) 2024-04-28T13:05:46.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 -- 172.21.15.53:0/1414786944 learned_addr learned my addr 172.21.15.53:0/1414786944 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:46.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0317fff640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0320102a60 0x7f032019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:46.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0325184640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0320103420 0x7f032019fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:47.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 -- 172.21.15.53:0/1414786944 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0320103420 msgr2=0x7f032019fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:47.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0320103420 0x7f032019fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:47.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 -- 172.21.15.53:0/1414786944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0320102a60 msgr2=0x7f032019d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:47.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0320102a60 0x7f032019d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:47.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 -- 172.21.15.53:0/1414786944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03201a0750 con 0x7f0320101870 2024-04-28T13:05:47.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0325184640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0320103420 0x7f032019fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T13:05:47.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0317fff640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0320102a60 0x7f032019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T13:05:47.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0324983640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320101870 0x7f032006a690 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f031c00c970 tx=0x7f031c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:47.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0315ffb640 1 -- 172.21.15.53:0/1414786944 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c007c10 con 0x7f0320101870 2024-04-28T13:05:47.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0315ffb640 1 -- 172.21.15.53:0/1414786944 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f031c007db0 con 0x7f0320101870 2024-04-28T13:05:47.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03201a0a40 con 0x7f0320101870 2024-04-28T13:05:47.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0315ffb640 1 -- 172.21.15.53:0/1414786944 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c00f870 con 0x7f0320101870 2024-04-28T13:05:47.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.995+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03201a0f90 con 0x7f0320101870 2024-04-28T13:05:47.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.996+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0320108b90 con 0x7f0320101870 2024-04-28T13:05:47.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.997+0000 7f0315ffb640 1 -- 172.21.15.53:0/1414786944 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f031c00fa10 con 0x7f0320101870 2024-04-28T13:05:47.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.998+0000 7f0315ffb640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f03080a3c40 0x7f03080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:47.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.998+0000 7f0317fff640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f03080a3c40 0x7f03080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:47.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.998+0000 7f0315ffb640 1 -- 172.21.15.53:0/1414786944 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8498+0+0 (secure 0 0 0) 0x7f031c0d0b40 con 0x7f0320101870 2024-04-28T13:05:47.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:46.999+0000 7f0317fff640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f03080a3c40 0x7f03080a6100 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f031002a930 tx=0x7f0310005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:47.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.003+0000 7f0315ffb640 1 -- 172.21.15.53:0/1414786944 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f031c09d2e0 con 0x7f0320101870 2024-04-28T13:05:47.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:46 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1940951253' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:05:47.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.188+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0320063530 con 0x7f0320101870 2024-04-28T13:05:47.201 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:46 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1940951253' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:05:47.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:46 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1940951253' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:05:47.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.869+0000 7f0315ffb640 1 -- 172.21.15.53:0/1414786944 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) ==== 181+0+0 (secure 0 0 0) 0x7f031c09cbe0 con 0x7f0320101870 2024-04-28T13:05:47.869 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' removed 2024-04-28T13:05:47.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f03080a3c40 msgr2=0x7f03080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:47.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f03080a3c40 0x7f03080a6100 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f031002a930 tx=0x7f0310005fd0 comp rx=0 tx=0).stop 2024-04-28T13:05:47.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320101870 msgr2=0x7f032006a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320101870 0x7f032006a690 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f031c00c970 tx=0x7f031c00ce30 comp rx=0 tx=0).stop 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 shutdown_connections 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f03080a3c40 0x7f03080a6100 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0320103420 0x7f032019fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0320102a60 0x7f032019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 --2- 172.21.15.53:0/1414786944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0320101870 0x7f032006a690 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 >> 172.21.15.53:0/1414786944 conn(0x7f03200fcde0 msgr2=0x7f0320105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:47.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 shutdown_connections 2024-04-28T13:05:47.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:47.873+0000 7f0326be7640 1 -- 172.21.15.53:0/1414786944 wait complete. 2024-04-28T13:05:47.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-28T13:05:48.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/2937892701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58101850 msgr2=0x7f9a58101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:48.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/2937892701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58101850 0x7f9a58101c70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9a48009900 tx=0x7f9a4802a410 comp rx=0 tx=0).stop 2024-04-28T13:05:48.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/2937892701 shutdown_connections 2024-04-28T13:05:48.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/2937892701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a58103400 0x7f9a58109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/2937892701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a58102a40 0x7f9a58102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/2937892701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58101850 0x7f9a58101c70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/2937892701 >> 172.21.15.53:0/2937892701 conn(0x7f9a580fcdc0 msgr2=0x7f9a580ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:48.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/2937892701 shutdown_connections 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/2937892701 wait complete. 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 Processor -- start 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.000+0000 7f9a5e3b6640 1 -- start start 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a58101850 0x7f9a5819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a58102a40 0x7f9a5819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58103400 0x7f9a581a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5810d350 con 0x7f9a58103400 2024-04-28T13:05:48.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5810d1d0 con 0x7f9a58102a40 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5810d4d0 con 0x7f9a58101850 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58103400 0x7f9a581a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58103400 0x7f9a581a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50982/0 (socket says 172.21.15.53:50982) 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 -- 172.21.15.53:0/3860045404 learned_addr learned my addr 172.21.15.53:0/3860045404 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a577fe640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a58102a40 0x7f9a5819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a57fff640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a58101850 0x7f9a5819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 -- 172.21.15.53:0/3860045404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a58101850 msgr2=0x7f9a5819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:48.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a58101850 0x7f9a5819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 -- 172.21.15.53:0/3860045404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a58102a40 msgr2=0x7f9a5819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:48.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a58102a40 0x7f9a5819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 -- 172.21.15.53:0/3860045404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a581a2a00 con 0x7f9a58103400 2024-04-28T13:05:48.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5c953640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58103400 0x7f9a581a2390 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009fd0 tx=0x7f9a5000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:48.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a557fa640 1 -- 172.21.15.53:0/3860045404 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50017070 con 0x7f9a58103400 2024-04-28T13:05:48.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a557fa640 1 -- 172.21.15.53:0/3860045404 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a50003e20 con 0x7f9a58103400 2024-04-28T13:05:48.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a581a2cf0 con 0x7f9a58103400 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a557fa640 1 -- 172.21.15.53:0/3860045404 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50010460 con 0x7f9a58103400 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.001+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a581a3190 con 0x7f9a58103400 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.002+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a581a33c0 con 0x7f9a58103400 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.003+0000 7f9a557fa640 1 -- 172.21.15.53:0/3860045404 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9a50003970 con 0x7f9a58103400 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.004+0000 7f9a557fa640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9a400a3c40 0x7f9a400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.004+0000 7f9a57fff640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9a400a3c40 0x7f9a400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.004+0000 7f9a557fa640 1 -- 172.21.15.53:0/3860045404 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 7944+0+0 (secure 0 0 0) 0x7f9a500d0270 con 0x7f9a58103400 2024-04-28T13:05:48.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.005+0000 7f9a57fff640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9a400a3c40 0x7f9a400a6100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9a48000c00 tx=0x7f9a480314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:05:48.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.009+0000 7f9a557fa640 1 -- 172.21.15.53:0/3860045404 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9a5009cbc0 con 0x7f9a58103400 2024-04-28T13:05:48.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: pgmap v805: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-28T13:05:48.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:05:48.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:05:48.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: Cluster is now healthy 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1940951253' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6832/3053379528,v1:172.21.15.77:6833/3053379528] up:boot 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6834/2000586345,v1:172.21.15.191:6835/2000586345] up:boot 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6834/2799525971,v1:172.21.15.77:6836/2799525971] up:boot 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6837/321021239,v1:172.21.15.191:6839/321021239] up:boot 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.191:6836/3512260269,v1:172.21.15.191:6838/3512260269] up:boot 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: mds.? [v2:172.21.15.77:6835/3031436469,v1:172.21.15.77:6837/3031436469] up:boot 2024-04-28T13:05:48.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: fsmap 12 up:standby 2024-04-28T13:05:48.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:05:48.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:05:48.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:05:48.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:05:48.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:05:48.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T13:05:48.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:47 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1414786944' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:05:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.200+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9a581a3bf0 con 0x7f9a58103400 2024-04-28T13:05:48.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: pgmap v805: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-28T13:05:48.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:05:48.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:05:48.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: Cluster is now healthy 2024-04-28T13:05:48.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1940951253' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:05:48.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6832/3053379528,v1:172.21.15.77:6833/3053379528] up:boot 2024-04-28T13:05:48.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6834/2000586345,v1:172.21.15.191:6835/2000586345] up:boot 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6834/2799525971,v1:172.21.15.77:6836/2799525971] up:boot 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6837/321021239,v1:172.21.15.191:6839/321021239] up:boot 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.191:6836/3512260269,v1:172.21.15.191:6838/3512260269] up:boot 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: mds.? [v2:172.21.15.77:6835/3031436469,v1:172.21.15.77:6837/3031436469] up:boot 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: fsmap 12 up:standby 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:05:48.203 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:05:48.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:05:48.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:05:48.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T13:05:48.204 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:47 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1414786944' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:05:48.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: pgmap v805: 97 pgs: 97 active+clean; 636 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-28T13:05:48.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:05:48.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:05:48.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: Cluster is now healthy 2024-04-28T13:05:48.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1940951253' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:05:48.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6832/3053379528,v1:172.21.15.77:6833/3053379528] up:boot 2024-04-28T13:05:48.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6834/2000586345,v1:172.21.15.191:6835/2000586345] up:boot 2024-04-28T13:05:48.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6834/2799525971,v1:172.21.15.77:6836/2799525971] up:boot 2024-04-28T13:05:48.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6837/321021239,v1:172.21.15.191:6839/321021239] up:boot 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.191:6836/3512260269,v1:172.21.15.191:6838/3512260269] up:boot 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: mds.? [v2:172.21.15.77:6835/3031436469,v1:172.21.15.77:6837/3031436469] up:boot 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: fsmap 12 up:standby 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:05:48.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='mgr.14152 172.21.15.53:0/1724450978' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T13:05:48.294 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:47 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1414786944' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:05:48.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.874+0000 7f9a557fa640 1 -- 172.21.15.53:0/3860045404 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) ==== 169+0+0 (secure 0 0 0) 0x7f9a5009c4c0 con 0x7f9a58103400 2024-04-28T13:05:48.874 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' removed 2024-04-28T13:05:48.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.877+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9a400a3c40 msgr2=0x7f9a400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:48.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.877+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9a400a3c40 0x7f9a400a6100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9a48000c00 tx=0x7f9a480314a0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.877+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58103400 msgr2=0x7f9a581a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:05:48.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.877+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58103400 0x7f9a581a2390 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009fd0 tx=0x7f9a5000b260 comp rx=0 tx=0).stop 2024-04-28T13:05:48.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 shutdown_connections 2024-04-28T13:05:48.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:6800/3118663054,v1:172.21.15.53:6801/3118663054] conn(0x7f9a400a3c40 0x7f9a400a6100 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a58103400 0x7f9a581a2390 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a58102a40 0x7f9a5819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 --2- 172.21.15.53:0/3860045404 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a58101850 0x7f9a5819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:05:48.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 >> 172.21.15.53:0/3860045404 conn(0x7f9a580fcdc0 msgr2=0x7f9a58069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:05:48.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 shutdown_connections 2024-04-28T13:05:48.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-28T13:05:48.878+0000 7f9a5e3b6640 1 -- 172.21.15.53:0/3860045404 wait complete. 2024-04-28T13:05:48.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:05:48.917 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:05:48.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:05:48.983 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-28T13:05:48.984 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:05:49.000 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:05:49.017 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:05:49.039 INFO:tasks.cephadm:Stopping all daemons... 2024-04-28T13:05:49.040 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-28T13:05:49.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a 2024-04-28T13:05:49.149 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:48 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/1414786944' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:05:49.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:48 smithi053 ceph-mon[29182]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T13:05:49.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:48 smithi053 ceph-mon[29182]: from='client.? 172.21.15.53:0/3860045404' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:05:49.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:49 smithi053 systemd[1]: Stopping Ceph mon.a for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:05:49.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:48 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/1414786944' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:05:49.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:48 smithi077 ceph-mon[32553]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T13:05:49.202 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:48 smithi077 ceph-mon[32553]: from='client.? 172.21.15.53:0/3860045404' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:05:49.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:48 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/1414786944' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:05:49.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:48 smithi191 ceph-mon[33016]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T13:05:49.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:48 smithi191 ceph-mon[33016]: from='client.? 172.21.15.53:0/3860045404' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:05:49.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a[29178]: 2024-04-28T13:05:49.379+0000 7fe261b5f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:05:49.650 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a[29178]: 2024-04-28T13:05:49.379+0000 7fe261b5f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-28T13:05:49.650 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:49 smithi053 podman[85135]: 2024-04-28 13:05:49.553798824 +0000 UTC m=+0.320786725 container died bd4ec601fb045968ff6b3b6e181d0d2f3baf1dfddc183383b0ec6218c7422e36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-28T13:05:50.625 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:50 smithi053 podman[85135]: 2024-04-28 13:05:50.154702037 +0000 UTC m=+0.921689934 container cleanup bd4ec601fb045968ff6b3b6e181d0d2f3baf1dfddc183383b0ec6218c7422e36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-28T13:05:50.625 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:50 smithi053 bash[85135]: ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a 2024-04-28T13:05:50.625 INFO:journalctl@ceph.mon.a.smithi053.stdout:Apr 28 13:05:50 smithi053 podman[85149]: 2024-04-28 13:05:50.381570388 +0000 UTC m=+0.826716263 container remove bd4ec601fb045968ff6b3b6e181d0d2f3baf1dfddc183383b0ec6218c7422e36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:05:50.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mon.a.service' 2024-04-28T13:05:50.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:05:50.697 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-28T13:05:50.697 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-28T13:05:50.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mon.b 2024-04-28T13:05:51.011 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:50 smithi077 systemd[1]: Stopping Ceph mon.b for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:05:51.452 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-b[32549]: 2024-04-28T13:05:51.010+0000 7f6e67284700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:05:51.452 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-b[32549]: 2024-04-28T13:05:51.010+0000 7f6e67284700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-28T13:05:51.452 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:51 smithi077 podman[67346]: 2024-04-28 13:05:51.195472741 +0000 UTC m=+0.305916362 container died 49b5bb26e08958be2dd70b2aaff7f092926282e6c5b1c08ec3a2fa18b6450981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-28T13:05:51.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:51 smithi077 podman[67346]: 2024-04-28 13:05:51.729090816 +0000 UTC m=+0.839534423 container cleanup 49b5bb26e08958be2dd70b2aaff7f092926282e6c5b1c08ec3a2fa18b6450981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-28T13:05:51.952 INFO:journalctl@ceph.mon.b.smithi077.stdout:Apr 28 13:05:51 smithi077 bash[67346]: ceph-10525726-055c-11ef-bc93-c7b262605968-mon-b 2024-04-28T13:05:52.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mon.b.service' 2024-04-28T13:05:52.249 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:05:52.249 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-28T13:05:52.249 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-28T13:05:52.250 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mon.c 2024-04-28T13:05:52.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:52 smithi191 systemd[1]: Stopping Ceph mon.c for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:05:52.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c[33012]: 2024-04-28T13:05:52.534+0000 7f03a98a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:05:52.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c[33012]: 2024-04-28T13:05:52.534+0000 7f03a98a1700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-28T13:05:52.791 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:52 smithi191 podman[68534]: 2024-04-28 13:05:52.627806619 +0000 UTC m=+0.187334720 container died 1490b39f630f1ef1ac41b364001085c2d6092b6ab2b13ddc3f6524bfd448b120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-28T13:05:53.291 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:53 smithi191 podman[68534]: 2024-04-28 13:05:53.129317646 +0000 UTC m=+0.688845735 container cleanup 1490b39f630f1ef1ac41b364001085c2d6092b6ab2b13ddc3f6524bfd448b120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-28T13:05:53.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 28 13:05:53 smithi191 bash[68534]: ceph-10525726-055c-11ef-bc93-c7b262605968-mon-c 2024-04-28T13:05:53.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mon.c.service' 2024-04-28T13:05:53.549 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:05:53.550 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-28T13:05:53.550 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-28T13:05:53.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.x 2024-04-28T13:05:53.899 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 13:05:53 smithi053 systemd[1]: Stopping Ceph mgr.x for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:05:54.326 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 13:05:54 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T13:05:54.000+0000 7f85a7dcb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:05:54.327 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 13:05:54 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x[29418]: 2024-04-28T13:05:54.000+0000 7f85a7dcb700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-28T13:05:54.327 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 13:05:54 smithi053 podman[85257]: 2024-04-28 13:05:54.033064616 +0000 UTC m=+0.282283758 container died 271ec44b4c367648752f2c44c211aa50cfcb7ffcc6237b26387a044ddfcbc5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:05:54.602 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 13:05:54 smithi053 podman[85257]: 2024-04-28 13:05:54.43529356 +0000 UTC m=+0.684512700 container cleanup 271ec44b4c367648752f2c44c211aa50cfcb7ffcc6237b26387a044ddfcbc5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-28T13:05:54.603 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 13:05:54 smithi053 bash[85257]: ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x 2024-04-28T13:05:54.853 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Apr 28 13:05:54 smithi053 podman[85271]: 2024-04-28 13:05:54.603445311 +0000 UTC m=+0.570282880 container remove 271ec44b4c367648752f2c44c211aa50cfcb7ffcc6237b26387a044ddfcbc5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-28T13:05:54.857 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-28T13:05:54.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:05:54.894 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-28T13:05:54.894 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-28T13:05:54.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.y 2024-04-28T13:05:55.201 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 13:05:54 smithi077 systemd[1]: Stopping Ceph mgr.y for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:05:55.510 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 13:05:55 smithi077 podman[67480]: 2024-04-28 13:05:55.23031541 +0000 UTC m=+0.148713830 container died 371a85653376411011afc9a4cf808690ed4cfb8519b77bbd7a479a299f69ff73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-28T13:05:55.885 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 13:05:55 smithi077 podman[67480]: 2024-04-28 13:05:55.601469583 +0000 UTC m=+0.519868005 container cleanup 371a85653376411011afc9a4cf808690ed4cfb8519b77bbd7a479a299f69ff73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-28T13:05:55.885 INFO:journalctl@ceph.mgr.y.smithi077.stdout:Apr 28 13:05:55 smithi077 bash[67480]: ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-y 2024-04-28T13:05:56.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-28T13:05:56.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:05:56.071 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-28T13:05:56.071 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-28T13:05:56.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.z 2024-04-28T13:05:56.414 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 13:05:56 smithi191 systemd[1]: Stopping Ceph mgr.z for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:05:56.703 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 13:05:56 smithi191 podman[68669]: 2024-04-28 13:05:56.41490825 +0000 UTC m=+0.155095164 container died 0f9a78af91f4b6eacec2ad826ac6cab2bd90e503cc17ced5d122479171ca0ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-28T13:05:57.041 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 13:05:56 smithi191 podman[68669]: 2024-04-28 13:05:56.794979905 +0000 UTC m=+0.535166870 container cleanup 0f9a78af91f4b6eacec2ad826ac6cab2bd90e503cc17ced5d122479171ca0ec5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-28T13:05:57.041 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 28 13:05:56 smithi191 bash[68669]: ceph-10525726-055c-11ef-bc93-c7b262605968-mgr-z 2024-04-28T13:05:57.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-28T13:05:57.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:05:57.300 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-28T13:05:57.300 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-28T13:05:57.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.0 2024-04-28T13:05:57.605 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:05:57 smithi053 systemd[1]: Stopping Ceph osd.0 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:05:57.899 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:05:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0[43553]: 2024-04-28T13:05:57.605+0000 7f72e0beb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:05:57.899 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:05:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0[43553]: 2024-04-28T13:05:57.605+0000 7f72e0beb700 -1 osd.0 91 *** Got signal Terminated *** 2024-04-28T13:05:57.899 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:05:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0[43553]: 2024-04-28T13:05:57.605+0000 7f72e0beb700 -1 osd.0 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:03.399 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:03 smithi053 podman[85382]: 2024-04-28 13:06:03.128421369 +0000 UTC m=+5.623432418 container died 804262456f6b817b503d1b2949146f5f677b4110a9ecd984c648bfc21c8f2f85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-28T13:06:03.895 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:03 smithi053 podman[85382]: 2024-04-28 13:06:03.636188489 +0000 UTC m=+6.131199537 container cleanup 804262456f6b817b503d1b2949146f5f677b4110a9ecd984c648bfc21c8f2f85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:06:03.895 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:03 smithi053 bash[85382]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0 2024-04-28T13:06:04.149 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:03 smithi053 podman[85399]: 2024-04-28 13:06:03.89607857 +0000 UTC m=+0.764169909 container remove 804262456f6b817b503d1b2949146f5f677b4110a9ecd984c648bfc21c8f2f85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-28T13:06:04.479 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:04 smithi053 podman[85489]: 2024-04-28 13:06:04.230296316 +0000 UTC m=+0.018609289 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:06:04.480 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:04 smithi053 podman[85489]: 2024-04-28 13:06:04.355609894 +0000 UTC m=+0.143922894 container create 990d9f35afdb52420ede1d91e95f9965380a1fc373e6fd789db78d4a88e82c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-28T13:06:04.899 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:04 smithi053 podman[85489]: 2024-04-28 13:06:04.600377717 +0000 UTC m=+0.388690686 container init 990d9f35afdb52420ede1d91e95f9965380a1fc373e6fd789db78d4a88e82c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-28T13:06:04.899 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:04 smithi053 podman[85489]: 2024-04-28 13:06:04.604407916 +0000 UTC m=+0.392720877 container start 990d9f35afdb52420ede1d91e95f9965380a1fc373e6fd789db78d4a88e82c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-28T13:06:04.900 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:04 smithi053 podman[85489]: 2024-04-28 13:06:04.666504006 +0000 UTC m=+0.454816982 container attach 990d9f35afdb52420ede1d91e95f9965380a1fc373e6fd789db78d4a88e82c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:05.228 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 28 13:06:04 smithi053 podman[85489]: 2024-04-28 13:06:04.911230612 +0000 UTC m=+0.699543615 container died 990d9f35afdb52420ede1d91e95f9965380a1fc373e6fd789db78d4a88e82c92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-28T13:06:05.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.0.service' 2024-04-28T13:06:05.568 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:05.568 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-28T13:06:05.569 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-28T13:06:05.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.1 2024-04-28T13:06:05.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:05 smithi077 systemd[1]: Stopping Ceph osd.1 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:06:06.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1[37141]: 2024-04-28T13:06:05.889+0000 7fdcaa779700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:06.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1[37141]: 2024-04-28T13:06:05.889+0000 7fdcaa779700 -1 osd.1 91 *** Got signal Terminated *** 2024-04-28T13:06:06.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1[37141]: 2024-04-28T13:06:05.889+0000 7fdcaa779700 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:11.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:11 smithi077 podman[67607]: 2024-04-28 13:06:11.251090484 +0000 UTC m=+5.485064265 container died d604ee695f4d1e52a90f100cd8196d54086003aeefee1089fed3c1bc48b93c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:11.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:11 smithi077 podman[67607]: 2024-04-28 13:06:11.665117587 +0000 UTC m=+5.899091375 container cleanup d604ee695f4d1e52a90f100cd8196d54086003aeefee1089fed3c1bc48b93c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:11.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:11 smithi077 bash[67607]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1 2024-04-28T13:06:12.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:11 smithi077 podman[67624]: 2024-04-28 13:06:11.908083784 +0000 UTC m=+0.660307140 container remove d604ee695f4d1e52a90f100cd8196d54086003aeefee1089fed3c1bc48b93c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:06:12.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:12 smithi077 podman[67714]: 2024-04-28 13:06:12.237497211 +0000 UTC m=+0.017818729 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:06:12.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:12 smithi077 podman[67714]: 2024-04-28 13:06:12.342059816 +0000 UTC m=+0.122381326 container create d0e65f573215862c4e82cd5b6536393643e93859270ecc0cb47c5dc9125e838f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-28T13:06:12.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:12 smithi077 podman[67714]: 2024-04-28 13:06:12.594912809 +0000 UTC m=+0.375234319 container init d0e65f573215862c4e82cd5b6536393643e93859270ecc0cb47c5dc9125e838f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-28T13:06:12.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:12 smithi077 podman[67714]: 2024-04-28 13:06:12.598232375 +0000 UTC m=+0.378553895 container start d0e65f573215862c4e82cd5b6536393643e93859270ecc0cb47c5dc9125e838f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-28T13:06:12.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:12 smithi077 podman[67714]: 2024-04-28 13:06:12.703433133 +0000 UTC m=+0.483754663 container attach d0e65f573215862c4e82cd5b6536393643e93859270ecc0cb47c5dc9125e838f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-28T13:06:13.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Apr 28 13:06:12 smithi077 podman[67714]: 2024-04-28 13:06:12.895871969 +0000 UTC m=+0.676193474 container died d0e65f573215862c4e82cd5b6536393643e93859270ecc0cb47c5dc9125e838f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-04-28T13:06:13.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.1.service' 2024-04-28T13:06:13.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:13.624 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-28T13:06:13.625 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-28T13:06:13.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.2 2024-04-28T13:06:14.016 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:13 smithi191 systemd[1]: Stopping Ceph osd.2 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:06:14.291 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2[38184]: 2024-04-28T13:06:14.015+0000 7f5350686700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:14.291 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2[38184]: 2024-04-28T13:06:14.015+0000 7f5350686700 -1 osd.2 91 *** Got signal Terminated *** 2024-04-28T13:06:14.291 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2[38184]: 2024-04-28T13:06:14.015+0000 7f5350686700 -1 osd.2 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:19.609 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:19 smithi191 podman[68798]: 2024-04-28 13:06:19.327194213 +0000 UTC m=+5.506431059 container died c9e2c686910d417fb3f10d8da6b85e0e18b3cc1c2b6eb3d5a1bd897da46122df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:06:19.976 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:19 smithi191 podman[68798]: 2024-04-28 13:06:19.707777833 +0000 UTC m=+5.887014684 container cleanup c9e2c686910d417fb3f10d8da6b85e0e18b3cc1c2b6eb3d5a1bd897da46122df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-28T13:06:19.976 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:19 smithi191 bash[68798]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2 2024-04-28T13:06:20.286 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:19 smithi191 podman[68815]: 2024-04-28 13:06:19.977077564 +0000 UTC m=+0.649884252 container remove c9e2c686910d417fb3f10d8da6b85e0e18b3cc1c2b6eb3d5a1bd897da46122df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-28T13:06:20.541 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:20 smithi191 podman[68911]: 2024-04-28 13:06:20.189846964 +0000 UTC m=+0.017794054 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:06:20.541 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:20 smithi191 podman[68911]: 2024-04-28 13:06:20.344420886 +0000 UTC m=+0.172368003 container create e95f8f2db320e1c5ed596f51008c727efbd4880ea2102df8466fc5a993dadc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-28T13:06:20.883 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:20 smithi191 podman[68911]: 2024-04-28 13:06:20.589071868 +0000 UTC m=+0.417018942 container init e95f8f2db320e1c5ed596f51008c727efbd4880ea2102df8466fc5a993dadc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-28T13:06:20.883 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:20 smithi191 podman[68911]: 2024-04-28 13:06:20.59292794 +0000 UTC m=+0.420875015 container start e95f8f2db320e1c5ed596f51008c727efbd4880ea2102df8466fc5a993dadc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-28T13:06:20.883 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:20 smithi191 podman[68911]: 2024-04-28 13:06:20.681224424 +0000 UTC m=+0.509171499 container attach e95f8f2db320e1c5ed596f51008c727efbd4880ea2102df8466fc5a993dadc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-04-28T13:06:21.234 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 28 13:06:20 smithi191 podman[68911]: 2024-04-28 13:06:20.883999193 +0000 UTC m=+0.711946268 container died e95f8f2db320e1c5ed596f51008c727efbd4880ea2102df8466fc5a993dadc71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-28T13:06:21.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.2.service' 2024-04-28T13:06:21.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:21.602 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-28T13:06:21.602 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-28T13:06:21.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.3 2024-04-28T13:06:21.899 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:21 smithi053 systemd[1]: Stopping Ceph osd.3 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:06:22.175 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:21 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T13:06:21.939+0000 7f98412f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:22.175 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:21 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T13:06:21.939+0000 7f98412f0700 -1 osd.3 91 *** Got signal Terminated *** 2024-04-28T13:06:22.175 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:21 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T13:06:21.939+0000 7f98412f0700 -1 osd.3 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:24.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:23.646+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:24.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:24 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:24.078+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:24.942 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:24.597+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:24.942 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:24.727+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:25.078 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:24 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T13:06:24.726+0000 7f983d8e4700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.989798+0000 front 2024-04-28T13:05:58.989651+0000 (oldest deadline 2024-04-28T13:06:24.289795+0000) 2024-04-28T13:06:25.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:25 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:25.079+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:25.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:25 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:25.045+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:26.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:25.746+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:26.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:25.576+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:26.063 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:25 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T13:06:25.680+0000 7f983d8e4700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.989798+0000 front 2024-04-28T13:05:58.989651+0000 (oldest deadline 2024-04-28T13:06:24.289795+0000) 2024-04-28T13:06:26.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:26 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:26.063+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:26.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:26 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:26.094+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:26.791 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:26.537+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:26.791 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:26.764+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:27.017 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:26 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3[50257]: 2024-04-28T13:06:26.683+0000 7f983d8e4700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.989798+0000 front 2024-04-28T13:05:58.989651+0000 (oldest deadline 2024-04-28T13:06:24.289795+0000) 2024-04-28T13:06:27.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:27.208+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:27.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:27.016+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:27.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:27 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:27.134+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:27.780 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:27 smithi053 podman[85596]: 2024-04-28 13:06:27.44569421 +0000 UTC m=+5.635032020 container died a490eade1186e8dec029522119d5cefaea6d328b26e5a81571f6f9fd93524592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:06:27.781 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:27 smithi053 podman[85596]: 2024-04-28 13:06:27.781491087 +0000 UTC m=+5.970828900 container cleanup a490eade1186e8dec029522119d5cefaea6d328b26e5a81571f6f9fd93524592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:28.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:27.743+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:28.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:27.580+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:28.149 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:27 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:27.997+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:28.149 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:27 smithi053 bash[85596]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3 2024-04-28T13:06:28.149 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:28 smithi053 podman[85626]: 2024-04-28 13:06:28.024427114 +0000 UTC m=+0.567186053 container remove a490eade1186e8dec029522119d5cefaea6d328b26e5a81571f6f9fd93524592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-28T13:06:28.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:28 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:28.088+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:28.598 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:28 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:28.223+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:28.599 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:28 smithi053 podman[85718]: 2024-04-28 13:06:28.467134902 +0000 UTC m=+0.109909043 container create f15294cb220bc14f96dd12a9bcadd4bb6c3b7e7a69581029ea3ff5520cf33287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:28.599 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:28 smithi053 podman[85718]: 2024-04-28 13:06:28.375631763 +0000 UTC m=+0.018405917 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:06:28.899 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:28 smithi053 podman[85718]: 2024-04-28 13:06:28.711871494 +0000 UTC m=+0.354645636 container init f15294cb220bc14f96dd12a9bcadd4bb6c3b7e7a69581029ea3ff5520cf33287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-28T13:06:28.899 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:28 smithi053 podman[85718]: 2024-04-28 13:06:28.715245233 +0000 UTC m=+0.358019381 container start f15294cb220bc14f96dd12a9bcadd4bb6c3b7e7a69581029ea3ff5520cf33287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-28T13:06:28.899 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:28 smithi053 podman[85718]: 2024-04-28 13:06:28.81136997 +0000 UTC m=+0.454144125 container attach f15294cb220bc14f96dd12a9bcadd4bb6c3b7e7a69581029ea3ff5520cf33287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2) 2024-04-28T13:06:29.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:28.722+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:29.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:28.605+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:29.211 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 28 13:06:29 smithi053 podman[85718]: 2024-04-28 13:06:29.028638788 +0000 UTC m=+0.671412941 container died f15294cb220bc14f96dd12a9bcadd4bb6c3b7e7a69581029ea3ff5520cf33287 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True) 2024-04-28T13:06:29.211 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:28 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:28.994+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:29.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:29 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:29.103+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:29.614 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:29 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:29.211+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:29.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.3.service' 2024-04-28T13:06:29.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:29.687 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-28T13:06:29.688 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-28T13:06:29.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.4 2024-04-28T13:06:30.010 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:29 smithi077 systemd[1]: Stopping Ceph osd.4 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:06:30.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:29.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:30.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:29.686+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:30.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:29 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:29.996+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:30.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:30 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:30.182+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:30.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:30 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:30.146+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:30.452 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:30 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4[44080]: 2024-04-28T13:06:30.010+0000 7f27ee9ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:30.452 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:30 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4[44080]: 2024-04-28T13:06:30.010+0000 7f27ee9ca700 -1 osd.4 91 *** Got signal Terminated *** 2024-04-28T13:06:30.452 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:30 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4[44080]: 2024-04-28T13:06:30.010+0000 7f27ee9ca700 -1 osd.4 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:31.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:30.689+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:31.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:30.664+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:31.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:31 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:31.155+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:31.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:30 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:30.968+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:31.451 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:31 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:31.132+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:32.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:31 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:31.689+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:32.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:31 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:31.707+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:32.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:32 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:32.134+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:32.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:31 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:31.997+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:32.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:32 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:32.167+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:33.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:32 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:32.743+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:33.042 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:32 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:32.643+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:33.042 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:32 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:32.643+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:33.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:33.046+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:33.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:33.177+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:33.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:33.177+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:33.451 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:33 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4[44080]: 2024-04-28T13:06:33.413+0000 7f27e6fb6700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:07.798755+0000 front 2024-04-28T13:06:07.798780+0000 (oldest deadline 2024-04-28T13:06:33.098920+0000) 2024-04-28T13:06:33.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:33 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:33.199+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:34.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:33 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:33.680+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:34.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:33 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:33.680+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:34.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:33 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:33.704+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:34.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:33 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:33.704+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:34.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:34 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:34.077+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:34.400 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:34 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:34.182+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:34.400 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:34 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:34.182+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:34.446 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:34 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:34.163+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:34.447 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:34 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:34.163+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:34.702 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:34 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4[44080]: 2024-04-28T13:06:34.446+0000 7f27e6fb6700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:07.798755+0000 front 2024-04-28T13:06:07.798780+0000 (oldest deadline 2024-04-28T13:06:33.098920+0000) 2024-04-28T13:06:35.001 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:34 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:34.715+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:35.001 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:34 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:34.715+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:35.001 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:34 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:34.659+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:35.002 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:34 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:34.659+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:35.291 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:35 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:35.194+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:35.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:35 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:35.041+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:35.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:35 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:35.041+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:35.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:35 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:35.218+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:35.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:35 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:35.218+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:35.451 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:35 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:35.198+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:35.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:35 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:35.198+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:35.744 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:35 smithi077 podman[67834]: 2024-04-28 13:06:35.456507065 +0000 UTC m=+5.567273945 container died 9463589fc6cd7091fd5654eafce565e6fde8d6a087f7eb92ad958931d325bf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-28T13:06:35.744 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:35 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:35.557+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:36.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:35 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:35.688+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:36.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:35 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:35.688+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:36.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:35 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:35.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:36.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:35 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:35.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:36.085 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:35 smithi077 podman[67834]: 2024-04-28 13:06:35.843721938 +0000 UTC m=+5.954488863 container cleanup 9463589fc6cd7091fd5654eafce565e6fde8d6a087f7eb92ad958931d325bf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-28T13:06:36.085 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:35 smithi077 bash[67834]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4 2024-04-28T13:06:36.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:36 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:36.065+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:36.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:36 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:36.065+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:36.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:36 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:36.237+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:36.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:36 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:36.238+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:36.411 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:36 smithi077 podman[67853]: 2024-04-28 13:06:36.086366755 +0000 UTC m=+0.628755615 container remove 9463589fc6cd7091fd5654eafce565e6fde8d6a087f7eb92ad958931d325bf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-28T13:06:36.411 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:36 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:36.237+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:36.412 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:36 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:36.237+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:36.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:36 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:36.207+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:36.673 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:36 smithi077 podman[67944]: 2024-04-28 13:06:36.314911106 +0000 UTC m=+0.018144436 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:06:36.673 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:36 smithi077 podman[67944]: 2024-04-28 13:06:36.458151142 +0000 UTC m=+0.161384461 container create 22fc679b78927101b26d661efaa8ccc72e8bd02c9a6cdb6d1556e311ac6f8d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:36.673 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:36 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:36.570+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:36.952 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:36 smithi077 podman[67944]: 2024-04-28 13:06:36.673796971 +0000 UTC m=+0.377030302 container init 22fc679b78927101b26d661efaa8ccc72e8bd02c9a6cdb6d1556e311ac6f8d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:36.952 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:36 smithi077 podman[67944]: 2024-04-28 13:06:36.677445958 +0000 UTC m=+0.380679278 container start 22fc679b78927101b26d661efaa8ccc72e8bd02c9a6cdb6d1556e311ac6f8d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-28T13:06:36.953 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:36 smithi077 podman[67944]: 2024-04-28 13:06:36.773340223 +0000 UTC m=+0.476573543 container attach 22fc679b78927101b26d661efaa8ccc72e8bd02c9a6cdb6d1556e311ac6f8d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-28T13:06:37.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:36 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:36.682+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:37.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:36 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:36.682+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:37.042 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:36 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:36.681+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:37.042 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:36 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:36.681+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:37.273 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 28 13:06:36 smithi077 podman[67944]: 2024-04-28 13:06:36.970351472 +0000 UTC m=+0.673584792 container died 22fc679b78927101b26d661efaa8ccc72e8bd02c9a6cdb6d1556e311ac6f8d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:37.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:37 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:37.223+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:37.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:37 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:37.223+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:37.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:37 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:37.090+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:37.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:37 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:37.090+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:37.538 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:37 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:37.273+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:37.539 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:37 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:37.273+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:37.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:37.249+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:37.611 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.4.service' 2024-04-28T13:06:37.648 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:37.648 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-28T13:06:37.648 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-28T13:06:37.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.5 2024-04-28T13:06:37.952 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:37 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:37.538+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:37.971 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:37.656+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:37.971 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:37.656+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:37.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:37.698+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:37.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:37.698+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:37.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:37 smithi191 systemd[1]: Stopping Ceph osd.5 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:06:37.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:37.970+0000 7f3ba528a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:37.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:37.970+0000 7f3ba528a700 -1 osd.5 91 *** Got signal Terminated *** 2024-04-28T13:06:37.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:37 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:37.970+0000 7f3ba528a700 -1 osd.5 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:38.291 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:38 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:38.203+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:38.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:38 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:38.129+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:38.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:38 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:38.129+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:38.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:38 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:38.189+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:38.400 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:38 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:38.189+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:38.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:38 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:38.505+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:38.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:38 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:38.279+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:38.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:38 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:38.279+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:39.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:38 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:38.663+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:39.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:38 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:38.663+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:39.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:38 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:38.672+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:39.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:38 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:38.672+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:39.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:39 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:39.113+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:39.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:39 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:39.113+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:39.400 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:39 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:39.150+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:39.400 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:39 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:39.150+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:39.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:39 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:39.172+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:39.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:39 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:39.464+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:39.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:39 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:39.306+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:39.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:39 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:39.306+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:40.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:39 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:39.699+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:40.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:39 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:39.699+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:40.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:39 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:39.638+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:40.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:39 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:39.638+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:40.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:40 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:40.191+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:40.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:40 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:40.191+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:40.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:40 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:40.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:40.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:40 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:40.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:40.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:40 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:40.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:40.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:40 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:40.170+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:40.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:40 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:40.170+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:40.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:40 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:40.453+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:40.703 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:40 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:40.453+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:40.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:40 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:40.297+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:40.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:40 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:40.297+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:41.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:40 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:40.736+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:41.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:40 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:40.736+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:41.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:40 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:40.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:41.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:40 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:40.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:41.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:40 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:40.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:41.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:41 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:41.239+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:41.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:41 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:41.239+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:41.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:41 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:41.094+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:41.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:41 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:41.094+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:41.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:41 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:41.094+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:41.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:41 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:41.168+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:41.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:41 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:41.168+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:41.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:41 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:41.465+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:41.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:41 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:41.465+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:41.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:41 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:41.330+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:41.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:41 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:41.330+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:41.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:41 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:41.330+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:42.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:41 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:41.770+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:42.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:41 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:41.770+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:42.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:41 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:41.630+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:42.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:41 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:41.630+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:42.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:41 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:41.630+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:42.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:42 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:42.246+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:42.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:42 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:42.246+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:42.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:42 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:42.050+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:42.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:42 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:42.050+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:42.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:42 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:42.050+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:42.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:42 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:42.178+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:42.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:42 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:42.178+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:42.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:42 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:42.461+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:42.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:42 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:42.461+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:42.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:42 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:42.293+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:42.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:42 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:42.293+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:42.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:42 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:42.293+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:43.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:42 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:42.763+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.382302+0000 front 2024-04-28T13:05:58.382160+0000 (oldest deadline 2024-04-28T13:06:24.282152+0000) 2024-04-28T13:06:43.041 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:42 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5[45137]: 2024-04-28T13:06:42.763+0000 7f3b9d876700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.483818+0000 front 2024-04-28T13:06:09.483944+0000 (oldest deadline 2024-04-28T13:06:31.783723+0000) 2024-04-28T13:06:43.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:42 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:42.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:43.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:42 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:42.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:43.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:42 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:42.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:43.265 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:43 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:43.003+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:43.265 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:43 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:43.003+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:43.265 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:43 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:43.003+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:43.541 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:43 smithi191 podman[69028]: 2024-04-28 13:06:43.371652283 +0000 UTC m=+5.526230643 container died 782073519f7e2e82d389f27753aac5a3c4e20ffcc9fbb745e2dad8f3362f62bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-28T13:06:43.542 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:43 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:43.150+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:43.542 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:43 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:43.150+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:43.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:43 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:43.265+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:43.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:43 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:43.265+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:43.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:43 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:43.265+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:43.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:43 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:43.455+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:43.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:43 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:43.455+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:43.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:43 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:43.266+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:43.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:43 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:43.266+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:43.703 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:43 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:43.266+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:44.030 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:43 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:43.619+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:44.031 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:43 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:43.619+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:44.031 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:43 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:43.619+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:44.031 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:43 smithi191 podman[69028]: 2024-04-28 13:06:43.746228518 +0000 UTC m=+5.900806876 container cleanup 782073519f7e2e82d389f27753aac5a3c4e20ffcc9fbb745e2dad8f3362f62bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-28T13:06:44.031 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:43 smithi191 bash[69028]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5 2024-04-28T13:06:44.291 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:44 smithi191 podman[69046]: 2024-04-28 13:06:44.031232942 +0000 UTC m=+0.656247885 container remove 782073519f7e2e82d389f27753aac5a3c4e20ffcc9fbb745e2dad8f3362f62bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:06:44.291 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:44 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:44.162+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:44.291 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:44 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:44.162+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:44.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:44 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:44.257+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:44.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:44 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:44.257+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:44.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:44 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:44.257+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:44.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:44 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:44.025+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:44.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:44 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:44.025+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:44.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:44 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:44.025+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:44.502 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:44 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:44.245+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:44.502 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:44 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:44.245+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:44.503 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:44 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:44.245+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:44.627 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:44 smithi191 podman[69139]: 2024-04-28 13:06:44.266644694 +0000 UTC m=+0.018400415 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:06:44.627 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:44 smithi191 podman[69139]: 2024-04-28 13:06:44.39928357 +0000 UTC m=+0.151039290 container create 40256cb792097e793ce41b360c448667a5e2a59582cfd09df416e6b34d93bd70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:06:44.952 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:44 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:44.503+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:44.952 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:44 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:44.503+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:44.980 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:44 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:44.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:44.981 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:44 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:44.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:44.981 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:44 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:44.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:44.981 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:44 smithi191 podman[69139]: 2024-04-28 13:06:44.67805216 +0000 UTC m=+0.429807873 container init 40256cb792097e793ce41b360c448667a5e2a59582cfd09df416e6b34d93bd70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-28T13:06:44.981 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:44 smithi191 podman[69139]: 2024-04-28 13:06:44.681445045 +0000 UTC m=+0.433200759 container start 40256cb792097e793ce41b360c448667a5e2a59582cfd09df416e6b34d93bd70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-28T13:06:44.981 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:44 smithi191 podman[69139]: 2024-04-28 13:06:44.752309341 +0000 UTC m=+0.504065065 container attach 40256cb792097e793ce41b360c448667a5e2a59582cfd09df416e6b34d93bd70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-28T13:06:44.982 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 28 13:06:44 smithi191 podman[69139]: 2024-04-28 13:06:44.981518874 +0000 UTC m=+0.733274590 container died 40256cb792097e793ce41b360c448667a5e2a59582cfd09df416e6b34d93bd70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-28T13:06:45.289 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:45.192+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:45.289 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:45.192+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:45.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:45.287+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:45.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:45.287+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:45.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:45.287+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:45.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:45.047+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:45.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:45.047+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:45.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:45.047+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:45.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.5.service' 2024-04-28T13:06:45.635 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:45.635 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-28T13:06:45.636 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-28T13:06:45.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.6 2024-04-28T13:06:45.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:45.470+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:45.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:45.470+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:45.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:45.236+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:45.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:45.236+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:45.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:45 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:45.236+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:45.942 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:45 smithi053 systemd[1]: Stopping Ceph osd.6 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:06:46.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:45.618+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:46.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:45.618+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:46.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:45 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:45.618+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:46.316 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:45.942+0000 7f6fcfb6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:46.316 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:45.942+0000 7f6fcfb6b700 -1 osd.6 91 *** Got signal Terminated *** 2024-04-28T13:06:46.317 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:45.942+0000 7f6fcfb6b700 -1 osd.6 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:46.317 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:46.096+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:46.317 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:46.096+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:46.317 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:46.096+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:46.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:46 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:46.239+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:46.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:46 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:46.239+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:46.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:46.316+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:46.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:46.316+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:46.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:46 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:46.316+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:46.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:46 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:46.445+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:46.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:46 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:46.445+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:46.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:46 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:46.219+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:46.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:46 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:46.219+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:46.702 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:46 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:46.219+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:47.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:46 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:46.619+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:47.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:46 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:46.619+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:47.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:46 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:46.619+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:47.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:47.287+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:47.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:47.287+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:47.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:47.287+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:47.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:47.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:47.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:47.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:47.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:47 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:47.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:47.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:47 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:47.419+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:47.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:47 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:47.419+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:47.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:47 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:47.172+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:47.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:47 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:47.172+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:47.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:47 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:47.172+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:47.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:47.274+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:47.541 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:47.274+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:48.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:47.630+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:48.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:47.630+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:48.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:47 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:47.630+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:48.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:48.314+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:48.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:48.314+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:48.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:48.314+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:48.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:48.086+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:48.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:48.086+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:48.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:48 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:48.086+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:48.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:48.413+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:48.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:48.413+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:48.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:48.179+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:48.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:48.179+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:48.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:48 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:48.179+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:48.657 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:48.315+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:48.658 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:48.315+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:48.658 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:48.315+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:49.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:48.656+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:49.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:48.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:49.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:48 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:48.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:49.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:49.324+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:49.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:49.324+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:49.400 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:49.324+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:49.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:49.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:49.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:49.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:49.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:49.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:49.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:49 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:49.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:49.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:49.384+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:49.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:49.384+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:49.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:49.177+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:49.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:49.177+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:49.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:49.177+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:49.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:49.177+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:49.629 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:49.334+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:49.630 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:49.334+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:49.630 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:49.334+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:50.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:49.629+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:50.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:49.629+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:50.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:49 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:49.629+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:50.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:50.413+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:50.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:50.413+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:50.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:50.413+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:50.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:50.196+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:50.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:50.196+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:50.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:50.196+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:50.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:50 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:50.196+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:50.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:50.331+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:06:02.293594+0000 front 2024-04-28T13:06:02.293627+0000 (oldest deadline 2024-04-28T13:06:26.393854+0000) 2024-04-28T13:06:50.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:50.331+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.395774+0000 front 2024-04-28T13:06:10.395749+0000 (oldest deadline 2024-04-28T13:06:32.695726+0000) 2024-04-28T13:06:50.649 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6[57280]: 2024-04-28T13:06:50.331+0000 7f6fc8157700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:18.298435+0000 front 2024-04-28T13:06:18.298290+0000 (oldest deadline 2024-04-28T13:06:42.398350+0000) 2024-04-28T13:06:50.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:50.150+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:50.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:50.150+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:50.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:50.150+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:50.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:50 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:50.150+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:50.791 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:50.581+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:50.791 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:50.581+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:50.792 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:50.581+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:50.792 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:50.334+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:50.792 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:50.334+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:50.792 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:50 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:50.334+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:51.399 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:51 smithi053 podman[85824]: 2024-04-28 13:06:51.313543883 +0000 UTC m=+5.471832787 container died 44db5a237551b920eed273f3ada227ba4bdf90a41062387aaea5f854927aa366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-28T13:06:51.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:51.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:51.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:51.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:51.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:51.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:51.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:51 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:51.128+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:51.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:51.367+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:51.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:51.367+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:51.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:51.367+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:51.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:51.182+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:51.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:51.182+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:51.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:51.182+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:51.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:51.182+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:51.626 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:51.365+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:51.627 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:51.365+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:51.627 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:51.365+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:51.959 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:51 smithi053 podman[85824]: 2024-04-28 13:06:51.67572508 +0000 UTC m=+5.834013985 container cleanup 44db5a237551b920eed273f3ada227ba4bdf90a41062387aaea5f854927aa366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:06:51.960 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:51 smithi053 bash[85824]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6 2024-04-28T13:06:52.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:51.626+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:52.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:51.626+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:52.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:51 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:51.626+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:52.284 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:51 smithi053 podman[85843]: 2024-04-28 13:06:51.960397702 +0000 UTC m=+0.643867040 container remove 44db5a237551b920eed273f3ada227ba4bdf90a41062387aaea5f854927aa366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-28T13:06:52.284 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:52.140+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:52.284 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:52.140+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:52.285 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:52.140+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:52.285 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:52 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:52.140+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:52.395 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:52.137+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:52.429 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:52.137+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:52.429 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:52.137+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:52.429 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:52.137+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:52.580 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:52 smithi053 podman[85943]: 2024-04-28 13:06:52.187845481 +0000 UTC m=+0.025836309 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:06:52.580 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:52 smithi053 podman[85943]: 2024-04-28 13:06:52.286162705 +0000 UTC m=+0.124153522 container create 9afe8511b5df0bf9183e998904f07b0e8454b6db85d1bfa06ec402036abe0316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:06:52.580 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:52 smithi053 podman[85943]: 2024-04-28 13:06:52.522239975 +0000 UTC m=+0.360230792 container init 9afe8511b5df0bf9183e998904f07b0e8454b6db85d1bfa06ec402036abe0316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-04-28T13:06:52.580 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:52 smithi053 podman[85943]: 2024-04-28 13:06:52.525620606 +0000 UTC m=+0.363611432 container start 9afe8511b5df0bf9183e998904f07b0e8454b6db85d1bfa06ec402036abe0316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:06:52.581 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:52 smithi053 podman[85943]: 2024-04-28 13:06:52.580479242 +0000 UTC m=+0.418470067 container attach 9afe8511b5df0bf9183e998904f07b0e8454b6db85d1bfa06ec402036abe0316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:06:52.657 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:52.369+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:52.657 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:52.369+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:52.657 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:52.369+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:52.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:52.395+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:52.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:52.395+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:52.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:52 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:52.395+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:53.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:52.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:53.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:52.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:53.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:52 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:52.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:53.119 INFO:journalctl@ceph.osd.6.smithi053.stdout:Apr 28 13:06:52 smithi053 podman[85943]: 2024-04-28 13:06:52.839542816 +0000 UTC m=+0.677533631 container died 9afe8511b5df0bf9183e998904f07b0e8454b6db85d1bfa06ec402036abe0316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-28T13:06:53.394 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:53.143+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:53.394 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:53.143+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:53.395 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:53.143+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:53.395 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:53.143+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:53.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:53 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:53.119+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:53.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:53 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:53.119+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:53.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:53 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:53.119+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:53.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:53 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:53.119+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:53.569 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.6.service' 2024-04-28T13:06:53.616 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:53.616 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-28T13:06:53.616 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-28T13:06:53.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.7 2024-04-28T13:06:53.664 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:53.395+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:53.664 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:53.395+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:53.664 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:53.395+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:53.678 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:53 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:53.348+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:53.679 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:53 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:53.348+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:53.679 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:53 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:53.348+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:53.937 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:53 smithi077 systemd[1]: Stopping Ceph osd.7 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:06:54.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:53 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:53.679+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:54.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:53 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:53.679+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:54.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:53 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:53.679+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:54.202 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:53.937+0000 7f1faf1f7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:54.202 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:53.937+0000 7f1faf1f7700 -1 osd.7 91 *** Got signal Terminated *** 2024-04-28T13:06:54.202 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:53 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:53.937+0000 7f1faf1f7700 -1 osd.7 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:06:54.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:54.139+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:54.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:54.139+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:54.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:54.139+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:54.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:54.139+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:54.462 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:54.360+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:54.462 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:54.360+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:54.462 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:54 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:54.360+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:54.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:54 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:54.167+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:54.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:54 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:54.167+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:54.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:54 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:54.167+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:54.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:54 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:54.167+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:54.660 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:54 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:54.337+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:54.660 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:54 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:54.337+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:54.661 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:54 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:54.337+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:54.942 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:54 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:54.661+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:54.942 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:54 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:54.661+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:54.942 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:54 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:54.661+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:55.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:55.380+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:55.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:55.380+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:55.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:55.380+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:55.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:55.166+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:55.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:55.166+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:55.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:55.166+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:55.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:55.166+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:55.634 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:55 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:55.372+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:55.635 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:55 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:55.372+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:55.635 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:55 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:55.372+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:55.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:55 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:55.159+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:55.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:55 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:55.159+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:55.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:55 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:55.159+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:55.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:55 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:55.159+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:56.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:55 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:55.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:56.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:55 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:55.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:56.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:55 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:55.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:56.396 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:56 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:56.125+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:56.396 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:56 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:56.125+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:56.396 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:56 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:56.125+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:56.397 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:56 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:56.125+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:56.627 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:56 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:56.353+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:56.627 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:56 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:56.353+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:56.627 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:56 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:56.353+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:56.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:56 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:56.169+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:56.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:56 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:56.169+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:56.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:56 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:56.169+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:56.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:56 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:56.169+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:56.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:56 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:56.396+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:56.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:56 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:56.396+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:56.702 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:56 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:56.396+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:57.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:56 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:56.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:57.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:56 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:56.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:57.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:56 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:56.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:57.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:57.145+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:57.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:57.145+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:57.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:57.145+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:57.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:57.145+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:57.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:57 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:57.145+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:06:57.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:57.140+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:57.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:57.140+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:57.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:57.140+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:57.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:57.140+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:57.453 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:57.355+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:57.453 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:57.355+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:57.453 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:57.355+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:57.453 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:57 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:57.355+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:30.135422+0000 front 2024-04-28T13:06:30.135475+0000 (oldest deadline 2024-04-28T13:06:57.136110+0000) 2024-04-28T13:06:57.648 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:57.327+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:57.649 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:57.327+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:57.649 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:57.327+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:58.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:57.649+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:58.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:57.649+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:58.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:57 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:57.649+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:58.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:58 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:58.123+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:58.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:58 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:58.123+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:58.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:58 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:58.123+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:58.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:58 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:58.123+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:58.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:58 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:58.123+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:06:58.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:58.349+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.832990+0000 front 2024-04-28T13:06:10.833074+0000 (oldest deadline 2024-04-28T13:06:35.533088+0000) 2024-04-28T13:06:58.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:58.349+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.533582+0000 front 2024-04-28T13:06:15.533689+0000 (oldest deadline 2024-04-28T13:06:39.633609+0000) 2024-04-28T13:06:58.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:58.349+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:26.035163+0000 front 2024-04-28T13:06:26.035258+0000 (oldest deadline 2024-04-28T13:06:50.135120+0000) 2024-04-28T13:06:58.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7[50767]: 2024-04-28T13:06:58.349+0000 7f1fa77e3700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:30.135422+0000 front 2024-04-28T13:06:30.135475+0000 (oldest deadline 2024-04-28T13:06:57.136110+0000) 2024-04-28T13:06:58.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:58.168+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:58.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:58.168+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:58.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:58.168+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:58.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:58.168+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:58.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:58 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:58.168+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:06:58.663 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:58 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:58.351+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:58.663 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:58 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:58.351+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:58.663 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:58 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:58.351+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:58.663 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:58 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:58.351+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:06:59.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:58 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:58.663+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:06:59.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:58 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:58.663+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:06:59.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:58 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:58.663+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:06:59.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:59.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:06:59.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:59.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:06:59.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:59.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:06:59.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:59.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:06:59.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:06:59 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:06:59.078+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:06:59.452 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:59 smithi077 podman[68048]: 2024-04-28 13:06:59.272796631 +0000 UTC m=+5.461725354 container died 9631fb5cfd183a51509ce715df624e83f8e4f45ad3a2e01d63b934150c2c1d38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-28T13:06:59.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:59 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:59.173+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:06:59.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:59 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:59.173+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:06:59.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:59 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:59.173+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:06:59.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:59 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:59.173+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:06:59.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:06:59 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:06:59.173+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:06:59.674 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:59.308+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:06:59.675 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:59.308+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:06:59.675 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:59.308+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:06:59.675 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:06:59.308+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:06:59.904 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:59 smithi077 podman[68048]: 2024-04-28 13:06:59.646103957 +0000 UTC m=+5.835032682 container cleanup 9631fb5cfd183a51509ce715df624e83f8e4f45ad3a2e01d63b934150c2c1d38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-04-28T13:06:59.905 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:59 smithi077 bash[68048]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7 2024-04-28T13:07:00.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:59.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:00.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:59.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:00.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:59.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:00.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:06:59 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:06:59.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:00.203 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:06:59 smithi077 podman[68076]: 2024-04-28 13:06:59.905296399 +0000 UTC m=+0.631747388 container remove 9631fb5cfd183a51509ce715df624e83f8e4f45ad3a2e01d63b934150c2c1d38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-28T13:07:00.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:00 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:00.124+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:00.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:00 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:00.124+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:00.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:00 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:00.124+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:00.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:00 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:00.124+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:00.204 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:00 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:00.124+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:00.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:00.056+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:00.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:00.056+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:00.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:00.056+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:00.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:00.056+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:00.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:00 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:00.056+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:00.567 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:07:00 smithi077 podman[68170]: 2024-04-28 13:07:00.169639163 +0000 UTC m=+0.017923385 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:00.567 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:07:00 smithi077 podman[68170]: 2024-04-28 13:07:00.298259164 +0000 UTC m=+0.146543419 container create baa9fd1729cf3d06b70ab1d78176a2b16d5975a056e0827248941b657280e3e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-28T13:07:00.567 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:07:00 smithi077 podman[68170]: 2024-04-28 13:07:00.56746084 +0000 UTC m=+0.415745052 container init baa9fd1729cf3d06b70ab1d78176a2b16d5975a056e0827248941b657280e3e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:00.649 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:00.331+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:07:00.649 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:00.331+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:07:00.649 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:00.331+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:07:00.650 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:00.331+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:07:00.866 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:07:00 smithi077 podman[68170]: 2024-04-28 13:07:00.570867204 +0000 UTC m=+0.419151417 container start baa9fd1729cf3d06b70ab1d78176a2b16d5975a056e0827248941b657280e3e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-28T13:07:00.866 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:07:00 smithi077 podman[68170]: 2024-04-28 13:07:00.675607075 +0000 UTC m=+0.523891304 container attach baa9fd1729cf3d06b70ab1d78176a2b16d5975a056e0827248941b657280e3e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:07:01.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:00.649+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:01.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:00.649+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:01.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:00.649+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:01.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:00 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:00.649+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:01.137 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 28 13:07:00 smithi077 podman[68170]: 2024-04-28 13:07:00.866991683 +0000 UTC m=+0.715275911 container died baa9fd1729cf3d06b70ab1d78176a2b16d5975a056e0827248941b657280e3e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-28T13:07:01.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:01 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:01.082+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:01.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:01 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:01.082+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:01.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:01 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:01.082+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:01.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:01 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:01.082+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:01.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:01 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:01.082+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:01.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:01 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:01.136+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:01.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:01 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:01.136+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:01.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:01 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:01.136+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:01.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:01 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:01.136+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:01.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:01 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:01.136+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:01.492 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.7.service' 2024-04-28T13:07:01.530 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:01.531 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-28T13:07:01.531 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-28T13:07:01.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.8 2024-04-28T13:07:01.656 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:01.380+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:07:01.656 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:01.380+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:07:01.656 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:01.380+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:07:01.656 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:01.380+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:07:01.656 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 systemd[1]: Stopping Ceph osd.8 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:02.041 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:01.817+0000 7fadd7dc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:02.041 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:01.817+0000 7fadd7dc8700 -1 osd.8 91 *** Got signal Terminated *** 2024-04-28T13:07:02.041 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:01.817+0000 7fadd7dc8700 -1 osd.8 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:02.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:01.656+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:02.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:01.656+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:02.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:01.656+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:02.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:01 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:01.656+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:02.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:02.100+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:02.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:02.100+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:02.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:02.100+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:02.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:02.100+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:02.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:02 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:02.100+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:02.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:02 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:02.102+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:02.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:02 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:02.102+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:02.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:02 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:02.102+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:02.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:02 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:02.102+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:02.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:02 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:02.102+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:02.674 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:02.372+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:07:02.675 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:02.372+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:07:02.675 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:02.372+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:07:02.675 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:02.372+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:07:03.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:02.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:03.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:02.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:03.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:02.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:03.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:02 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:02.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:03.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:03 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:03.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:03.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:03 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:03.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:03.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:03 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:03.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:03.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:03 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:03.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:03.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:03 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:03.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:03.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:03 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:03.072+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:03.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:03 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:03.072+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:03.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:03 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:03.072+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:03.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:03 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:03.072+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:03.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:03 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:03.072+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:03.690 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:03.350+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:07:03.690 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:03.350+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:07:03.690 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:03.350+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:07:03.690 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:03.350+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:07:04.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:03.690+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:04.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:03.690+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:04.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:03.690+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:04.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:03.690+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:04.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:04.084+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:04.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:04.084+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:04.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:04.084+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:04.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:04.084+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:04.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:04 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:04.084+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:04.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:04 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:04.091+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:04.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:04 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:04.091+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:04.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:04 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:04.091+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:04.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:04 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:04.091+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:04.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:04 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:04.091+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:04.699 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:04.394+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:07:04.700 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:04.394+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:07:04.700 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:04.394+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:07:04.700 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:04.394+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:07:05.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:04.700+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:05.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:04.700+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:05.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:04.700+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:05.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:04.700+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:05.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:04 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:04.700+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:05.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:05.068+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:05.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:05.068+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:05.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:05.068+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:05.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:05.068+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:05.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:05.068+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:05.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:05 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:05.068+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:05.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:05.082+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:05.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:05.082+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:05.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:05.082+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:05.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:05.082+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:05.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:05 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:05.082+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:05.706 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:05.402+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:07:05.706 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:05.402+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:07:05.707 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:05.402+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:07:05.707 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:05.402+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:07:06.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:05.706+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:06.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:05.706+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:06.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:05.706+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:06.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:05.706+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:06.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:05 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:05.706+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:06.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:06.035+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:06.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:06.035+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:06.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:06.035+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:06.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:06.035+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:06.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:06.035+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:06.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:06 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:06.035+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:06.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:06 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:06.074+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:06.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:06 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:06.074+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:06.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:06 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:06.074+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:06.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:06 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:06.074+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:06.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:06 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:06.074+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:06.681 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:06.428+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.702968+0000 front 2024-04-28T13:06:10.702994+0000 (oldest deadline 2024-04-28T13:06:34.203082+0000) 2024-04-28T13:07:06.681 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:06.428+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:14.203367+0000 front 2024-04-28T13:06:14.203474+0000 (oldest deadline 2024-04-28T13:06:40.103514+0000) 2024-04-28T13:07:06.681 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:06.428+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:25.404384+0000 front 2024-04-28T13:06:25.404449+0000 (oldest deadline 2024-04-28T13:06:47.704463+0000) 2024-04-28T13:07:06.682 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8[51813]: 2024-04-28T13:07:06.428+0000 7fadd03b4700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.005643+0000 front 2024-04-28T13:06:33.005625+0000 (oldest deadline 2024-04-28T13:06:57.606044+0000) 2024-04-28T13:07:07.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:06.681+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:07.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:06.681+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:07.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:06.681+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:07.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:06.681+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:07.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:06 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:06.681+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:07.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:07 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:07.026+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:07.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:07 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:07.026+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:07.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:07 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:07.026+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:07.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:07 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:07.026+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:07.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:07 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:07.026+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:07.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:07 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:07.026+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:07.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:07 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:07.064+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:07.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:07 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:07.064+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:07.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:07 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:07.064+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:07.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:07 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:07.064+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:07.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:07 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:07.064+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:07.608 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:07 smithi191 podman[69257]: 2024-04-28 13:07:07.305005028 +0000 UTC m=+5.582934894 container died df4fe2ed32dd71bd0a4e099dce77d7093f1fdfc0eebfc0ff64bc1b67cc13f0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:07:07.900 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:07 smithi191 podman[69257]: 2024-04-28 13:07:07.699645945 +0000 UTC m=+5.977575860 container cleanup df4fe2ed32dd71bd0a4e099dce77d7093f1fdfc0eebfc0ff64bc1b67cc13f0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-28T13:07:07.900 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:07 smithi191 bash[69257]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8 2024-04-28T13:07:07.901 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:07 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:07.658+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:07.901 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:07 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:07.658+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:07.901 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:07 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:07.658+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:07.901 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:07 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:07.658+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:07.901 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:07 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:07.658+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:08.291 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:07 smithi191 podman[69274]: 2024-04-28 13:07:07.90082559 +0000 UTC m=+0.588762452 container remove df4fe2ed32dd71bd0a4e099dce77d7093f1fdfc0eebfc0ff64bc1b67cc13f0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:08.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:08 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:08.067+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:08.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:08 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:08.067+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:08.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:08 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:08.067+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:08.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:08 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:08.067+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:08.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:08 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:08.067+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:08.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:08 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:08.067+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:08.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.031+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:08.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.031+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:08.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.031+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:08.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.031+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:08.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.031+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:08.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.031+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:08.625 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:08 smithi191 podman[69364]: 2024-04-28 13:07:08.223163093 +0000 UTC m=+0.018074293 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:08.626 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:08 smithi191 podman[69364]: 2024-04-28 13:07:08.326619417 +0000 UTC m=+0.121530594 container create 85093038c379c7e6f4421735ad4c71eec566857e62a7d24adf1e924b391fdabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:07:08.626 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:08 smithi191 podman[69364]: 2024-04-28 13:07:08.554596668 +0000 UTC m=+0.349507846 container init 85093038c379c7e6f4421735ad4c71eec566857e62a7d24adf1e924b391fdabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:07:08.626 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:08 smithi191 podman[69364]: 2024-04-28 13:07:08.557767667 +0000 UTC m=+0.352678851 container start 85093038c379c7e6f4421735ad4c71eec566857e62a7d24adf1e924b391fdabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-28T13:07:09.041 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:08 smithi191 podman[69364]: 2024-04-28 13:07:08.671083918 +0000 UTC m=+0.465995107 container attach 85093038c379c7e6f4421735ad4c71eec566857e62a7d24adf1e924b391fdabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-28T13:07:09.042 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 28 13:07:08 smithi191 podman[69364]: 2024-04-28 13:07:08.849827568 +0000 UTC m=+0.644738749 container died 85093038c379c7e6f4421735ad4c71eec566857e62a7d24adf1e924b391fdabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:09.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:08 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:08.625+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:09.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:08 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:08.625+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:09.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:08 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:08.625+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:09.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:08 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:08.625+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:09.043 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:08 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:08.625+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:09.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:09.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:09.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:09.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:09.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:09.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.108+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:09.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.988+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:09.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.988+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:09.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.988+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:09.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.988+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:09.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.988+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:09.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:08 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:08.988+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:09.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.8.service' 2024-04-28T13:07:09.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:09.503 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-28T13:07:09.503 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-28T13:07:09.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.9 2024-04-28T13:07:09.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:09 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:09.650+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:09.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:09 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:09.650+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:09.715 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:09 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:09.650+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:09.715 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:09 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:09.650+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:09.715 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:09 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:09.650+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:09.844 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 systemd[1]: Stopping Ceph osd.9 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:10.149 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.844+0000 7fd20dd04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.844+0000 7fd20dd04700 -1 osd.9 91 *** Got signal Terminated *** 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:09 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:09.844+0000 7fd20dd04700 -1 osd.9 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:10 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:10.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:10 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:10.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:10 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:10.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:10 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:10.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:10 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:10.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:10.150 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:10 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:10.061+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:10.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:09 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:09.972+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:10.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:09 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:09.972+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:10.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:09 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:09.972+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:10.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:09 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:09.972+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:10.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:09 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:09.972+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:10.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:09 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:09.972+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:11.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:10 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:10.651+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:11.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:10 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:10.651+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:11.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:10 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:10.651+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:11.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:10 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:10.651+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:11.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:10 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:10.651+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:11.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:11.102+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:11.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:11.102+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:11.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:11.102+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:11.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:11.102+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:11.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:11.102+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:11.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:11 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:11.102+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:11.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:10 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:10.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:11.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:10 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:10.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:11.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:10 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:10.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:11.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:10 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:10.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:11.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:10 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:10.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:11.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:10 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:10.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:12.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:11 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:11.652+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:12.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:11 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:11.652+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:12.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:11 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:11.652+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:12.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:11 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:11.652+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:12.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:11 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:11.652+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:12.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:11 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:11.917+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:12.204 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:11 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:11.917+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:12.204 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:11 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:11.917+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:12.204 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:11 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:11.917+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:12.204 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:11 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:11.917+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:12.205 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:11 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:11.917+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:12.205 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:11 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:11.917+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:12.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:12 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:12.117+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:12.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:12 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:12.117+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:12.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:12 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:12.117+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:12.399 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:12 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:12.117+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:12.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:12 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:12.117+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:12.400 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:12 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:12.117+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:13.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:12 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:12.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:13.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:12 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:12.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:13.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:12 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:12.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:13.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:12 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:12.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:13.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:12 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:12.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:13.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:12 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:12.922+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:13.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:12 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:12.922+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:13.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:12 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:12.922+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:13.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:12 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:12.922+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:13.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:12 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:12.922+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:13.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:12 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:12.922+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:13.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:12 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:12.922+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:13.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:13 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:13.160+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:13.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:13 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:13.160+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:13.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:13 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:13.160+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:13.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:13 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:13.160+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:13.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:13 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:13.160+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:13.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:13 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:13.160+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:13.651 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:13 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:13.160+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:49.862898+0000 front 2024-04-28T13:06:49.862940+0000 (oldest deadline 2024-04-28T13:07:12.162876+0000) 2024-04-28T13:07:14.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:13 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:13.643+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:14.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:13 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:13.643+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:14.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:13 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:13.643+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:14.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:13 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:13.643+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:14.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:13 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:13.643+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:14.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:13 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:13.938+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:14.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:13 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:13.938+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:14.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:13 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:13.938+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:14.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:13 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:13.938+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:14.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:13 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:13.938+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:14.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:13 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:13.938+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:14.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:13 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:13.938+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:14.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:14 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:14.166+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.556500+0000 front 2024-04-28T13:05:59.556530+0000 (oldest deadline 2024-04-28T13:06:24.256713+0000) 2024-04-28T13:07:14.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:14 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:14.166+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:10.658315+0000 front 2024-04-28T13:06:10.658423+0000 (oldest deadline 2024-04-28T13:06:34.158200+0000) 2024-04-28T13:07:14.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:14 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:14.166+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:16.459501+0000 front 2024-04-28T13:06:16.459485+0000 (oldest deadline 2024-04-28T13:06:39.959167+0000) 2024-04-28T13:07:14.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:14 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:14.166+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.660100+0000 front 2024-04-28T13:06:24.660231+0000 (oldest deadline 2024-04-28T13:06:48.160258+0000) 2024-04-28T13:07:14.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:14 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:14.166+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:31.061312+0000 front 2024-04-28T13:06:31.061581+0000 (oldest deadline 2024-04-28T13:06:56.961188+0000) 2024-04-28T13:07:14.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:14 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:14.166+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:41.062129+0000 front 2024-04-28T13:06:41.062082+0000 (oldest deadline 2024-04-28T13:07:04.562078+0000) 2024-04-28T13:07:14.650 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:14 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9[64296]: 2024-04-28T13:07:14.166+0000 7fd2062f0700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:49.862898+0000 front 2024-04-28T13:06:49.862940+0000 (oldest deadline 2024-04-28T13:07:12.162876+0000) 2024-04-28T13:07:15.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:14.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:15.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:14.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:15.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:14.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:15.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:14.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:15.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:14.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:15.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:14.635+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:15.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:14 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:14.970+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:15.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:14 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:14.970+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:15.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:14 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:14.970+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:15.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:14 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:14.970+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:15.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:14 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:14.970+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:15.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:14 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:14.970+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:15.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:14 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:14.970+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:15.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:15 smithi053 podman[86050]: 2024-04-28 13:07:15.328332881 +0000 UTC m=+5.610655510 container died 67fb2e8494ed40fa18890fe24409e61bb09e0f0c0a1d60e5e44f2c5e157d9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-28T13:07:15.953 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:15 smithi053 podman[86050]: 2024-04-28 13:07:15.769235177 +0000 UTC m=+6.051557794 container cleanup 67fb2e8494ed40fa18890fe24409e61bb09e0f0c0a1d60e5e44f2c5e157d9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-28T13:07:15.953 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:15 smithi053 bash[86050]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9 2024-04-28T13:07:16.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:15 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:15.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:16.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:15 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:15.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:16.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:15 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:15.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:16.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:15 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:15.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:16.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:15 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:15.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:16.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:15 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:15.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:16.397 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:15 smithi053 podman[86068]: 2024-04-28 13:07:15.953714002 +0000 UTC m=+0.624361164 container remove 67fb2e8494ed40fa18890fe24409e61bb09e0f0c0a1d60e5e44f2c5e157d9108 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-28T13:07:16.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:15 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:15.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:16.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:15 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:15.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:16.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:15 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:15.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:16.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:15 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:15.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:16.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:15 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:15.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:16.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:15 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:15.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:16.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:15 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:15.955+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:16.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:16 smithi053 podman[86160]: 2024-04-28 13:07:16.301586216 +0000 UTC m=+0.018615965 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:16.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:16 smithi053 podman[86160]: 2024-04-28 13:07:16.404401031 +0000 UTC m=+0.121430779 container create 0b61943c59a5f5001a0438a97649d74c168b5c4bea488250946d219835643134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-28T13:07:16.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:16 smithi053 podman[86160]: 2024-04-28 13:07:16.62376599 +0000 UTC m=+0.340795754 container init 0b61943c59a5f5001a0438a97649d74c168b5c4bea488250946d219835643134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:07:16.649 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:16 smithi053 podman[86160]: 2024-04-28 13:07:16.627291622 +0000 UTC m=+0.344321370 container start 0b61943c59a5f5001a0438a97649d74c168b5c4bea488250946d219835643134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-28T13:07:16.937 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:16 smithi053 podman[86160]: 2024-04-28 13:07:16.706751279 +0000 UTC m=+0.423781030 container attach 0b61943c59a5f5001a0438a97649d74c168b5c4bea488250946d219835643134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-28T13:07:17.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:16 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:16.573+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:17.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:16 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:16.573+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:17.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:16 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:16.573+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:17.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:16 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:16.573+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:17.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:16 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:16.573+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:17.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:16 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:16.573+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:17.209 INFO:journalctl@ceph.osd.9.smithi053.stdout:Apr 28 13:07:16 smithi053 podman[86160]: 2024-04-28 13:07:16.93763895 +0000 UTC m=+0.654668714 container died 0b61943c59a5f5001a0438a97649d74c168b5c4bea488250946d219835643134 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:17.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:16 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:16.991+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:17.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:16 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:16.991+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:17.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:16 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:16.991+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:17.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:16 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:16.991+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:17.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:16 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:16.991+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:17.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:16 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:16.991+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:17.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:16 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:16.991+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:17.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.9.service' 2024-04-28T13:07:17.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:17.561 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-28T13:07:17.561 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-28T13:07:17.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.10 2024-04-28T13:07:17.856 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 systemd[1]: Stopping Ceph osd.10 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:18.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:17 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:17.575+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:18.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:17 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:17.575+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:18.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:17 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:17.575+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:18.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:17 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:17.575+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:18.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:17 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:17.575+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:18.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:17 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:17.575+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:18.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.856+0000 7f72aefca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:18.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.856+0000 7f72aefca700 -1 osd.10 91 *** Got signal Terminated *** 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.856+0000 7f72aefca700 -1 osd.10 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.947+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.947+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.947+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.947+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.947+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.947+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:18.203 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:17 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:17.947+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:19.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:18 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:18.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:19.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:18 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:18.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:19.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:18 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:18.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:19.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:18 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:18.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:19.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:18 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:18.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:19.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:18 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:18.606+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:19.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:18.982+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:19.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:18.982+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:19.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:18.982+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:19.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:18.982+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:19.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:18.983+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:19.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:18.983+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:19.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:18 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:18.983+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:20.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:19 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:19.654+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:20.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:19 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:19.654+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:20.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:19 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:19.654+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:20.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:19 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:19.654+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:20.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:19 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:19.654+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:20.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:19 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:19.654+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:20.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:19 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:19.654+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:20.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:20 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:20.007+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:20.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:20 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:20.007+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:20.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:20 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:20.007+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:20.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:20 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:20.007+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:20.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:20 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:20.007+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:20.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:20 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:20.007+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:20.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:20 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:20.007+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:21.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:20 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:20.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:21.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:20 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:20.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:21.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:20 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:20.627+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:21.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:20 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:20.628+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:21.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:20 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:20.628+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:21.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:20 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:20.628+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:21.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:20 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:20.628+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:21.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:21 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:21.018+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:21.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:21 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:21.018+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:21.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:21 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:21.018+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:21.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:21 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:21.018+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:21.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:21 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:21.018+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:21.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:21 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:21.018+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:21.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:21 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:21.018+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:22.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:21.610+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:22.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:21.610+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:22.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:21.610+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:22.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:21.610+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:22.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:21.610+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:22.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:21.610+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:22.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:21 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:21.610+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:22.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:22 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:22.028+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:59.323332+0000 front 2024-04-28T13:05:59.323350+0000 (oldest deadline 2024-04-28T13:06:24.023294+0000) 2024-04-28T13:07:22.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:22 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:22.028+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.324050+0000 front 2024-04-28T13:06:09.324082+0000 (oldest deadline 2024-04-28T13:06:34.024205+0000) 2024-04-28T13:07:22.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:22 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:22.028+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:15.724972+0000 front 2024-04-28T13:06:15.725012+0000 (oldest deadline 2024-04-28T13:06:41.025173+0000) 2024-04-28T13:07:22.452 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:22 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:22.028+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2024-04-28T13:06:24.426966+0000 front 2024-04-28T13:06:24.426838+0000 (oldest deadline 2024-04-28T13:06:49.126755+0000) 2024-04-28T13:07:22.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:22 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:22.028+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:32.027678+0000 front 2024-04-28T13:06:32.027696+0000 (oldest deadline 2024-04-28T13:06:57.327768+0000) 2024-04-28T13:07:22.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:22 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:22.028+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:42.529097+0000 front 2024-04-28T13:06:42.529214+0000 (oldest deadline 2024-04-28T13:07:07.229008+0000) 2024-04-28T13:07:22.453 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:22 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10[57359]: 2024-04-28T13:07:22.028+0000 7f72a75b6700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:47.229769+0000 front 2024-04-28T13:06:47.229834+0000 (oldest deadline 2024-04-28T13:07:11.329483+0000) 2024-04-28T13:07:23.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:22 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:22.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:23.119 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:22 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:22.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:23.119 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:22 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:22.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:23.119 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:22 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:22.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:23.119 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:22 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:22.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:23.120 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:22 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:22.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:23.120 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:22 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:22.642+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:23.614 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:23 smithi077 podman[68277]: 2024-04-28 13:07:23.326281793 +0000 UTC m=+5.571722346 container died 215b1baa3a4f3ed943573bc571bd6edaf11e24f493ad7778f23f665aa63d5b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:23.948 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:23 smithi077 podman[68277]: 2024-04-28 13:07:23.714519939 +0000 UTC m=+5.959960492 container cleanup 215b1baa3a4f3ed943573bc571bd6edaf11e24f493ad7778f23f665aa63d5b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:23.948 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:23 smithi077 bash[68277]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10 2024-04-28T13:07:24.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:23.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:24.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:23.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:24.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:23.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:24.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:23.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:24.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:23.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:24.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:23.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:24.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:23 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:23.688+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:24.202 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:23 smithi077 podman[68294]: 2024-04-28 13:07:23.94863867 +0000 UTC m=+0.619813974 container remove 215b1baa3a4f3ed943573bc571bd6edaf11e24f493ad7778f23f665aa63d5b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-28T13:07:24.501 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:24 smithi077 podman[68387]: 2024-04-28 13:07:24.181313545 +0000 UTC m=+0.017830894 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:24.501 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:24 smithi077 podman[68387]: 2024-04-28 13:07:24.307860133 +0000 UTC m=+0.144377510 container create 5680b06e9b99e5e547fac8b0f420bf9ab8f581cdc8af648f03f5a9ba6d3ab59b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-28T13:07:24.501 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:24 smithi077 podman[68387]: 2024-04-28 13:07:24.501589207 +0000 UTC m=+0.338106544 container init 5680b06e9b99e5e547fac8b0f420bf9ab8f581cdc8af648f03f5a9ba6d3ab59b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:07:24.801 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:24 smithi077 podman[68387]: 2024-04-28 13:07:24.50491927 +0000 UTC m=+0.341436605 container start 5680b06e9b99e5e547fac8b0f420bf9ab8f581cdc8af648f03f5a9ba6d3ab59b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:07:24.802 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:24 smithi077 podman[68387]: 2024-04-28 13:07:24.593259552 +0000 UTC m=+0.429776910 container attach 5680b06e9b99e5e547fac8b0f420bf9ab8f581cdc8af648f03f5a9ba6d3ab59b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:24.943 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:24.686+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:24.943 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:24.686+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:24.943 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:24.686+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:24.943 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:24.686+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:24.943 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:24.686+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:24.944 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:24.686+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:24.944 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:24 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:24.686+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:25.095 INFO:journalctl@ceph.osd.10.smithi077.stdout:Apr 28 13:07:24 smithi077 podman[68387]: 2024-04-28 13:07:24.802186485 +0000 UTC m=+0.638703816 container died 5680b06e9b99e5e547fac8b0f420bf9ab8f581cdc8af648f03f5a9ba6d3ab59b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-28T13:07:25.375 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.10.service' 2024-04-28T13:07:25.412 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:25.412 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-28T13:07:25.412 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-28T13:07:25.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@osd.11 2024-04-28T13:07:25.541 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 systemd[1]: Stopping Ceph osd.11 for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:26.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:26.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:26.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:26.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:26.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:26.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:26.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.678+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:26.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.717+0000 7f6c89f5d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:26.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.717+0000 7f6c89f5d700 -1 osd.11 91 *** Got signal Terminated *** 2024-04-28T13:07:26.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:25 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:25.717+0000 7f6c89f5d700 -1 osd.11 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:27.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:26.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:27.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:26.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:27.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:26.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:27.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:26.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:27.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:26.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:27.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:26.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:27.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:26 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:26.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:28.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:27.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:28.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:27.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:28.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:27.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:28.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:27.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:28.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:27.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:28.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:27.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:28.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:27 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:27.637+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:29.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:28.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:29.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:28.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:29.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:28.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:29.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:28.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:29.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:28.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:29.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:28.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:29.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:28 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:28.634+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:30.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:29.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:30.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:29.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:30.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:29.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:30.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:29.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:30.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:29.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:30.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:29.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:30.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:29 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:29.657+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:31.041 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-28T13:05:58.556357+0000 front 2024-04-28T13:05:58.556121+0000 (oldest deadline 2024-04-28T13:06:23.256130+0000) 2024-04-28T13:07:31.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6804 osd.1 since back 2024-04-28T13:06:09.157570+0000 front 2024-04-28T13:06:09.157323+0000 (oldest deadline 2024-04-28T13:06:33.257293+0000) 2024-04-28T13:07:31.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-28T13:06:17.358639+0000 front 2024-04-28T13:06:17.358852+0000 (oldest deadline 2024-04-28T13:06:40.258774+0000) 2024-04-28T13:07:31.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6812 osd.4 since back 2024-04-28T13:06:33.760986+0000 front 2024-04-28T13:06:33.761020+0000 (oldest deadline 2024-04-28T13:06:59.661035+0000) 2024-04-28T13:07:31.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-28T13:06:39.661359+0000 front 2024-04-28T13:06:39.661380+0000 (oldest deadline 2024-04-28T13:07:03.761691+0000) 2024-04-28T13:07:31.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2024-04-28T13:06:48.462749+0000 front 2024-04-28T13:06:48.462655+0000 (oldest deadline 2024-04-28T13:07:13.762685+0000) 2024-04-28T13:07:31.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6820 osd.7 since back 2024-04-28T13:06:56.063751+0000 front 2024-04-28T13:06:56.063781+0000 (oldest deadline 2024-04-28T13:07:19.563731+0000) 2024-04-28T13:07:31.042 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:30 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11[58847]: 2024-04-28T13:07:30.675+0000 7f6c86551700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-04-28T13:07:05.365244+0000 front 2024-04-28T13:07:05.365226+0000 (oldest deadline 2024-04-28T13:07:30.065287+0000) 2024-04-28T13:07:31.468 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:31 smithi191 podman[69492]: 2024-04-28 13:07:31.198943315 +0000 UTC m=+5.590073260 container died 420bd1cea5705eb81962bda9a02bb9c87c9c36cae06477b25a3017a5802c6fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-28T13:07:31.791 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:31 smithi191 podman[69492]: 2024-04-28 13:07:31.559243662 +0000 UTC m=+5.950373612 container cleanup 420bd1cea5705eb81962bda9a02bb9c87c9c36cae06477b25a3017a5802c6fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:07:31.791 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:31 smithi191 bash[69492]: ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11 2024-04-28T13:07:32.135 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:31 smithi191 podman[69512]: 2024-04-28 13:07:31.818781227 +0000 UTC m=+0.620003138 container remove 420bd1cea5705eb81962bda9a02bb9c87c9c36cae06477b25a3017a5802c6fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-04-28T13:07:32.429 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:32 smithi191 podman[69605]: 2024-04-28 13:07:32.039658525 +0000 UTC m=+0.018514442 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:32.429 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:32 smithi191 podman[69605]: 2024-04-28 13:07:32.136190201 +0000 UTC m=+0.115046106 container create 5111b65966e3f9842e128a9d5b3ef7bc254a4bc806388eb41f71804f96d580f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:07:32.430 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:32 smithi191 podman[69605]: 2024-04-28 13:07:32.355183205 +0000 UTC m=+0.334039117 container init 5111b65966e3f9842e128a9d5b3ef7bc254a4bc806388eb41f71804f96d580f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:32.430 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:32 smithi191 podman[69605]: 2024-04-28 13:07:32.358188423 +0000 UTC m=+0.337044329 container start 5111b65966e3f9842e128a9d5b3ef7bc254a4bc806388eb41f71804f96d580f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-28T13:07:32.430 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:32 smithi191 podman[69605]: 2024-04-28 13:07:32.430013757 +0000 UTC m=+0.408869672 container attach 5111b65966e3f9842e128a9d5b3ef7bc254a4bc806388eb41f71804f96d580f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-28T13:07:32.791 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:32 smithi191 podman[69605]: 2024-04-28 13:07:32.650994961 +0000 UTC m=+0.629850871 container died 5111b65966e3f9842e128a9d5b3ef7bc254a4bc806388eb41f71804f96d580f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:33.170 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 28 13:07:33 smithi191 podman[69630]: 2024-04-28 13:07:33.1500556 +0000 UTC m=+0.493511753 container remove 5111b65966e3f9842e128a9d5b3ef7bc254a4bc806388eb41f71804f96d580f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-28T13:07:33.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@osd.11.service' 2024-04-28T13:07:33.211 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:33.212 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-28T13:07:33.212 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-28T13:07:33.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.a 2024-04-28T13:07:33.554 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 13:07:33 smithi053 systemd[1]: Stopping Ceph mds.a for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:33.899 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 13:07:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a[71272]: 2024-04-28T13:07:33.553+0000 7f80409dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:33.899 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 13:07:33 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a[71272]: 2024-04-28T13:07:33.553+0000 7f80409dd700 -1 mds.a *** got signal Terminated *** 2024-04-28T13:07:35.928 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 13:07:35 smithi053 podman[86277]: 2024-04-28 13:07:35.659225224 +0000 UTC m=+2.238204530 container died 9bc0ae78e38689613db5523a4295e8120a8e2a2a334e70e8e7151573fe93e674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:36.295 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 13:07:36 smithi053 podman[86277]: 2024-04-28 13:07:36.011758279 +0000 UTC m=+2.590737564 container cleanup 9bc0ae78e38689613db5523a4295e8120a8e2a2a334e70e8e7151573fe93e674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-28T13:07:36.295 INFO:journalctl@ceph.mds.a.smithi053.stdout:Apr 28 13:07:36 smithi053 bash[86277]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-a 2024-04-28T13:07:36.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.a.service' 2024-04-28T13:07:36.487 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:36.487 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-28T13:07:36.488 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-28T13:07:36.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.d 2024-04-28T13:07:36.815 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:36 smithi053 systemd[1]: Stopping Ceph mds.d for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:37.149 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:36 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d[70318]: 2024-04-28T13:07:36.815+0000 7f5899bf4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:37.149 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:36 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d[70318]: 2024-04-28T13:07:36.815+0000 7f5899bf4700 -1 mds.d *** got signal Terminated *** 2024-04-28T13:07:39.616 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:39 smithi053 podman[86397]: 2024-04-28 13:07:39.360705277 +0000 UTC m=+2.651782653 container died 97fb01b2bc921be3971ff161c2a80f8fe7350102b550d12401a2bb854d4b4d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-28T13:07:39.891 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:39 smithi053 podman[86397]: 2024-04-28 13:07:39.699057876 +0000 UTC m=+2.990135259 container cleanup 97fb01b2bc921be3971ff161c2a80f8fe7350102b550d12401a2bb854d4b4d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:39.891 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:39 smithi053 bash[86397]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d 2024-04-28T13:07:40.141 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:39 smithi053 podman[86411]: 2024-04-28 13:07:39.891768426 +0000 UTC m=+0.532753549 container remove 97fb01b2bc921be3971ff161c2a80f8fe7350102b550d12401a2bb854d4b4d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-04-28T13:07:40.141 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:40 smithi053 systemd[1]: ceph-10525726-055c-11ef-bc93-c7b262605968@mds.d.service: Deactivated successfully. 2024-04-28T13:07:40.141 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:40 smithi053 systemd[1]: Stopped Ceph mds.d for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T13:07:40.141 INFO:journalctl@ceph.mds.d.smithi053.stdout:Apr 28 13:07:40 smithi053 systemd[1]: ceph-10525726-055c-11ef-bc93-c7b262605968@mds.d.service: Consumed 1.141s CPU time. 2024-04-28T13:07:40.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.d.service' 2024-04-28T13:07:40.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:40.176 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-28T13:07:40.176 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-28T13:07:40.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.g 2024-04-28T13:07:40.399 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:40 smithi053 systemd[1]: Stopping Ceph mds.g for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:40.899 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:40 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g[70915]: 2024-04-28T13:07:40.502+0000 7f7c2018d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:40.899 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:40 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g[70915]: 2024-04-28T13:07:40.502+0000 7f7c2018d700 -1 mds.g *** got signal Terminated *** 2024-04-28T13:07:44.844 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:44 smithi053 podman[86517]: 2024-04-28 13:07:44.516875862 +0000 UTC m=+4.112372436 container died ce9d304d4420a152597eea303d5bf8d24fa9889fb19de0aabe21c1ea65168be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-28T13:07:45.144 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:44 smithi053 podman[86517]: 2024-04-28 13:07:44.943841911 +0000 UTC m=+4.539338484 container cleanup ce9d304d4420a152597eea303d5bf8d24fa9889fb19de0aabe21c1ea65168be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-28T13:07:45.144 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:44 smithi053 bash[86517]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g 2024-04-28T13:07:45.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.g.service' 2024-04-28T13:07:45.399 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:45 smithi053 podman[86531]: 2024-04-28 13:07:45.1448162 +0000 UTC m=+0.629741954 container remove ce9d304d4420a152597eea303d5bf8d24fa9889fb19de0aabe21c1ea65168be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-28T13:07:45.399 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:45 smithi053 systemd[1]: ceph-10525726-055c-11ef-bc93-c7b262605968@mds.g.service: Deactivated successfully. 2024-04-28T13:07:45.399 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:45 smithi053 systemd[1]: Stopped Ceph mds.g for 10525726-055c-11ef-bc93-c7b262605968. 2024-04-28T13:07:45.399 INFO:journalctl@ceph.mds.g.smithi053.stdout:Apr 28 13:07:45 smithi053 systemd[1]: ceph-10525726-055c-11ef-bc93-c7b262605968@mds.g.service: Consumed 1.153s CPU time. 2024-04-28T13:07:45.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:45.415 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-28T13:07:45.416 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-28T13:07:45.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.j 2024-04-28T13:07:45.738 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 13:07:45 smithi053 systemd[1]: Stopping Ceph mds.j for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:46.149 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 13:07:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j[69930]: 2024-04-28T13:07:45.738+0000 7f7bb7fad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:46.149 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 13:07:45 smithi053 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j[69930]: 2024-04-28T13:07:45.738+0000 7f7bb7fad700 -1 mds.j *** got signal Terminated *** 2024-04-28T13:07:48.649 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 13:07:48 smithi053 podman[86637]: 2024-04-28 13:07:48.367918705 +0000 UTC m=+2.724171074 container died 534b2f156e3af79d1adaa3c7c3f2e494046be293838e5b7d8b399abfeef69bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:48.991 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 13:07:48 smithi053 podman[86637]: 2024-04-28 13:07:48.672325439 +0000 UTC m=+3.028577804 container cleanup 534b2f156e3af79d1adaa3c7c3f2e494046be293838e5b7d8b399abfeef69bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-28T13:07:48.991 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 13:07:48 smithi053 bash[86637]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j 2024-04-28T13:07:48.991 INFO:journalctl@ceph.mds.j.smithi053.stdout:Apr 28 13:07:48 smithi053 podman[86651]: 2024-04-28 13:07:48.873274644 +0000 UTC m=+0.495965923 container remove 534b2f156e3af79d1adaa3c7c3f2e494046be293838e5b7d8b399abfeef69bb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:07:49.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.j.service' 2024-04-28T13:07:49.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:49.046 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-28T13:07:49.047 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-28T13:07:49.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.b 2024-04-28T13:07:49.355 INFO:journalctl@ceph.mds.b.smithi077.stdout:Apr 28 13:07:49 smithi077 systemd[1]: Stopping Ceph mds.b for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:49.702 INFO:journalctl@ceph.mds.b.smithi077.stdout:Apr 28 13:07:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-b[63035]: 2024-04-28T13:07:49.354+0000 7f6460e80700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:49.702 INFO:journalctl@ceph.mds.b.smithi077.stdout:Apr 28 13:07:49 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-b[63035]: 2024-04-28T13:07:49.354+0000 7f6460e80700 -1 mds.b *** got signal Terminated *** 2024-04-28T13:07:50.797 INFO:journalctl@ceph.mds.b.smithi077.stdout:Apr 28 13:07:50 smithi077 podman[68490]: 2024-04-28 13:07:50.531856232 +0000 UTC m=+1.301623458 container died 5a6a25b915fb3a77107c6a444acaa4816bba42f36aa3be780a7d656cd4710c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-28T13:07:51.181 INFO:journalctl@ceph.mds.b.smithi077.stdout:Apr 28 13:07:50 smithi077 podman[68490]: 2024-04-28 13:07:50.897296582 +0000 UTC m=+1.667063816 container cleanup 5a6a25b915fb3a77107c6a444acaa4816bba42f36aa3be780a7d656cd4710c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:51.181 INFO:journalctl@ceph.mds.b.smithi077.stdout:Apr 28 13:07:50 smithi077 bash[68490]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-b 2024-04-28T13:07:51.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.b.service' 2024-04-28T13:07:51.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:51.363 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-28T13:07:51.363 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-28T13:07:51.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.e 2024-04-28T13:07:51.702 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:07:51 smithi077 systemd[1]: Stopping Ceph mds.e for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:52.202 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:07:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e[62416]: 2024-04-28T13:07:51.724+0000 7f73ec57f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:52.202 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:07:51 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e[62416]: 2024-04-28T13:07:51.724+0000 7f73ec57f700 -1 mds.e *** got signal Terminated *** 2024-04-28T13:07:54.184 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:07:53 smithi077 podman[68610]: 2024-04-28 13:07:53.882983957 +0000 UTC m=+2.290170711 container died e1f58deb45467bc35e69cc5b381fa75420ae6e2e28c7915e06d5c5ae24d1a394 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-28T13:07:54.452 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:07:54 smithi077 podman[68610]: 2024-04-28 13:07:54.301372428 +0000 UTC m=+2.708559193 container cleanup e1f58deb45467bc35e69cc5b381fa75420ae6e2e28c7915e06d5c5ae24d1a394 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-28T13:07:54.452 INFO:journalctl@ceph.mds.e.smithi077.stdout:Apr 28 13:07:54 smithi077 bash[68610]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-e 2024-04-28T13:07:54.667 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.e.service' 2024-04-28T13:07:54.699 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:54.699 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-28T13:07:54.700 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-28T13:07:54.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.h 2024-04-28T13:07:55.071 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:07:54 smithi077 systemd[1]: Stopping Ceph mds.h for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:55.452 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:07:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h[62723]: 2024-04-28T13:07:55.070+0000 7fd2db85d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:55.452 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:07:55 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h[62723]: 2024-04-28T13:07:55.070+0000 7fd2db85d700 -1 mds.h *** got signal Terminated *** 2024-04-28T13:07:58.188 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:07:57 smithi077 podman[68731]: 2024-04-28 13:07:57.887605596 +0000 UTC m=+2.965568570 container died b8b012781aba50f01a86cb0ae8c9e2d633a3ef67124a89fd55a428fe0aa32d88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-28T13:07:58.452 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:07:58 smithi077 podman[68731]: 2024-04-28 13:07:58.262691724 +0000 UTC m=+3.340654689 container cleanup b8b012781aba50f01a86cb0ae8c9e2d633a3ef67124a89fd55a428fe0aa32d88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:58.452 INFO:journalctl@ceph.mds.h.smithi077.stdout:Apr 28 13:07:58 smithi077 bash[68731]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-h 2024-04-28T13:07:58.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.h.service' 2024-04-28T13:07:58.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:58.721 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-28T13:07:58.721 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-28T13:07:58.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.k 2024-04-28T13:07:59.202 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:07:58 smithi077 systemd[1]: Stopping Ceph mds.k for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:07:59.202 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:07:59 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k[62100]: 2024-04-28T13:07:59.048+0000 7fbb3bf08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:59.202 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:07:59 smithi077 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k[62100]: 2024-04-28T13:07:59.048+0000 7fbb3bf08700 -1 mds.k *** got signal Terminated *** 2024-04-28T13:08:02.166 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:08:01 smithi077 podman[68850]: 2024-04-28 13:08:01.888942775 +0000 UTC m=+2.947326402 container died 231e8a50d16b4eab6a2dc09b47eedc558e598715e8123bf1e22f30f8a44b8ea5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:02.452 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:08:02 smithi077 podman[68850]: 2024-04-28 13:08:02.249912733 +0000 UTC m=+3.308296395 container cleanup 231e8a50d16b4eab6a2dc09b47eedc558e598715e8123bf1e22f30f8a44b8ea5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:08:02.453 INFO:journalctl@ceph.mds.k.smithi077.stdout:Apr 28 13:08:02 smithi077 bash[68850]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-k 2024-04-28T13:08:02.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.k.service' 2024-04-28T13:08:02.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:02.746 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-28T13:08:02.747 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-28T13:08:02.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.c 2024-04-28T13:08:03.032 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:08:02 smithi191 systemd[1]: Stopping Ceph mds.c for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:08:03.287 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:08:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c[63883]: 2024-04-28T13:08:03.032+0000 7f01c4b0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:03.288 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:08:03 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c[63883]: 2024-04-28T13:08:03.032+0000 7f01c4b0e700 -1 mds.c *** got signal Terminated *** 2024-04-28T13:08:06.174 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:08:05 smithi191 podman[69710]: 2024-04-28 13:08:05.887716057 +0000 UTC m=+2.954862707 container died 3cb94f0816f63f21bc8564eb3786b9e22273848bd305265f9afefe8013ba601d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:06.523 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:08:06 smithi191 podman[69710]: 2024-04-28 13:08:06.264995046 +0000 UTC m=+3.332141706 container cleanup 3cb94f0816f63f21bc8564eb3786b9e22273848bd305265f9afefe8013ba601d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-28T13:08:06.524 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 28 13:08:06 smithi191 bash[69710]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-c 2024-04-28T13:08:06.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.c.service' 2024-04-28T13:08:06.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:06.726 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-28T13:08:06.726 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-28T13:08:06.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.f 2024-04-28T13:08:07.041 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:08:06 smithi191 systemd[1]: Stopping Ceph mds.f for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:08:07.541 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:08:07 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f[63570]: 2024-04-28T13:08:07.185+0000 7fdd7eb5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:07.541 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:08:07 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f[63570]: 2024-04-28T13:08:07.185+0000 7fdd7eb5c700 -1 mds.f *** got signal Terminated *** 2024-04-28T13:08:10.177 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:08:09 smithi191 podman[69844]: 2024-04-28 13:08:09.891070206 +0000 UTC m=+2.938228989 container died 01f68a3a61a2c66e885d220ff1fca0a1f094b67b2d866ea3c2982e15d5347925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-28T13:08:10.502 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:08:10 smithi191 podman[69844]: 2024-04-28 13:08:10.277045719 +0000 UTC m=+3.324204518 container cleanup 01f68a3a61a2c66e885d220ff1fca0a1f094b67b2d866ea3c2982e15d5347925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-28T13:08:10.503 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 28 13:08:10 smithi191 bash[69844]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-f 2024-04-28T13:08:10.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.f.service' 2024-04-28T13:08:10.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:10.651 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-28T13:08:10.651 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-28T13:08:10.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.i 2024-04-28T13:08:10.791 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:08:10 smithi191 systemd[1]: Stopping Ceph mds.i for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:08:11.291 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:08:10 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: 2024-04-28T13:08:10.980+0000 7f246e0b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:11.291 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:08:10 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i[64193]: 2024-04-28T13:08:10.980+0000 7f246e0b9700 -1 mds.i *** got signal Terminated *** 2024-04-28T13:08:14.041 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:08:13 smithi191 podman[69965]: 2024-04-28 13:08:13.880903631 +0000 UTC m=+3.015350427 container died f60ee5b15a98dc8ab258c226d77fa14551d85cb55d64602f3366c1435cc7405d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-28T13:08:14.497 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:08:14 smithi191 podman[69965]: 2024-04-28 13:08:14.248612965 +0000 UTC m=+3.383059760 container cleanup f60ee5b15a98dc8ab258c226d77fa14551d85cb55d64602f3366c1435cc7405d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:08:14.497 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 28 13:08:14 smithi191 bash[69965]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-i 2024-04-28T13:08:14.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.i.service' 2024-04-28T13:08:14.658 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:14.658 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-28T13:08:14.659 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-28T13:08:14.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-10525726-055c-11ef-bc93-c7b262605968@mds.l 2024-04-28T13:08:14.791 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 13:08:14 smithi191 systemd[1]: Stopping Ceph mds.l for 10525726-055c-11ef-bc93-c7b262605968... 2024-04-28T13:08:15.291 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 13:08:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l[63251]: 2024-04-28T13:08:14.974+0000 7fdc42d24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:15.291 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 13:08:14 smithi191 ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l[63251]: 2024-04-28T13:08:14.974+0000 7fdc42d24700 -1 mds.l *** got signal Terminated *** 2024-04-28T13:08:18.693 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 13:08:18 smithi191 podman[70086]: 2024-04-28 13:08:18.41960009 +0000 UTC m=+3.547391345 container died 6196d3c0dcf5604f34c63b93ffb5d17f959b91e0678458d14a63de6903d06e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-28T13:08:19.041 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 13:08:18 smithi191 podman[70086]: 2024-04-28 13:08:18.783956933 +0000 UTC m=+3.911748181 container cleanup 6196d3c0dcf5604f34c63b93ffb5d17f959b91e0678458d14a63de6903d06e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:08:19.041 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 28 13:08:18 smithi191 bash[70086]: ceph-10525726-055c-11ef-bc93-c7b262605968-mds-l 2024-04-28T13:08:19.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10525726-055c-11ef-bc93-c7b262605968@mds.l.service' 2024-04-28T13:08:19.246 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:19.246 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-28T13:08:19.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10525726-055c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T13:08:19.447 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T13:08:21.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10525726-055c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T13:08:22.161 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T13:08:24.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10525726-055c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T13:08:24.686 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T13:08:26.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:08:27.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:08:27.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:08:27.078 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-28T13:08:27.080 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi053/crash 2024-04-28T13:08:27.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash -- . 2024-04-28T13:08:27.123 INFO:teuthology.orchestra.run.smithi053.stderr:tar: /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T13:08:27.123 INFO:teuthology.orchestra.run.smithi053.stderr:tar: Error is not recoverable: exiting now 2024-04-28T13:08:27.125 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi077/crash 2024-04-28T13:08:27.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash -- . 2024-04-28T13:08:27.216 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T13:08:27.216 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-04-28T13:08:27.218 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi191/crash 2024-04-28T13:08:27.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash -- . 2024-04-28T13:08:27.259 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/10525726-055c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T13:08:27.260 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-04-28T13:08:27.260 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-28T13:08:27.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-28T13:08:27.303 INFO:tasks.cephadm:Compressing logs... 2024-04-28T13:08:27.304 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T13:08:27.347 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T13:08:27.349 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T13:08:27.371 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T13:08:27.372 INFO:teuthology.orchestra.run.smithi053.stderr:: No such file or directory 2024-04-28T13:08:27.372 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log 2024-04-28T13:08:27.381 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T13:08:27.381 INFO:teuthology.orchestra.run.smithi191.stderr:: No such file or directory 2024-04-28T13:08:27.381 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log 2024-04-28T13:08:27.381 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-28T13:08:27.381 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-28T13:08:27.381 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr: 89.1% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T13:08:27.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.z.log: 90.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T13:08:27.385 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T13:08:27.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-28T13:08:27.387 INFO:teuthology.orchestra.run.smithi053.stderr: 90.3%/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.a.log: -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T13:08:27.387 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T13:08:27.387 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.7% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-28T13:08:27.387 INFO:teuthology.orchestra.run.smithi053.stderr: 94.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-28T13:08:27.387 INFO:teuthology.orchestra.run.smithi053.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T13:08:27.391 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T13:08:27.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-28T13:08:27.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log: 93.2% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-28T13:08:27.391 INFO:teuthology.orchestra.run.smithi191.stderr: 82.5% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T13:08:27.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-28T13:08:27.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T13:08:27.397 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-28T13:08:27.397 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T13:08:27.397 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.a.log: 82.8% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T13:08:27.397 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-28T13:08:27.397 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log: --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-28T13:08:27.397 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-28T13:08:27.397 INFO:teuthology.orchestra.run.smithi053.stderr: 94.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-28T13:08:27.398 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:: No such file or directory 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.7.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-28T13:08:27.399 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.audit.log.gz/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.h.log: 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.5% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T13:08:27.400 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T13:08:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.e.log: 94.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-28T13:08:27.401 INFO:teuthology.orchestra.run.smithi077.stderr: 93.0% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-28T13:08:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-28T13:08:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-28T13:08:27.402 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.8.log: /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-28T13:08:27.405 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.9.log: 91.1% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T13:08:27.408 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T13:08:27.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-28T13:08:27.428 INFO:teuthology.orchestra.run.smithi191.stderr: 91.0% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T13:08:27.432 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.1% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-28T13:08:27.432 INFO:teuthology.orchestra.run.smithi191.stderr: 94.2% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-28T13:08:27.452 INFO:teuthology.orchestra.run.smithi077.stderr: 94.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-28T13:08:27.455 INFO:teuthology.orchestra.run.smithi191.stderr: 94.5% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-28T13:08:27.462 INFO:teuthology.orchestra.run.smithi077.stderr: 94.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-28T13:08:27.565 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-28T13:08:27.709 INFO:teuthology.orchestra.run.smithi053.stderr: 90.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-28T13:08:27.714 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-28T13:08:27.735 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-28T13:08:27.866 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-28T13:08:27.971 INFO:teuthology.orchestra.run.smithi053.stderr: 92.6% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-28T13:08:28.047 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-28T13:08:28.112 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-28T13:08:28.351 INFO:teuthology.orchestra.run.smithi053.stderr: 93.2% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-28T13:08:28.369 INFO:teuthology.orchestra.run.smithi053.stderr: 90.3% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-28T13:08:28.470 INFO:teuthology.orchestra.run.smithi191.stderr: 93.2% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-28T13:08:28.522 INFO:teuthology.orchestra.run.smithi077.stderr: 93.2% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-28T13:08:28.562 INFO:teuthology.orchestra.run.smithi077.stderr: 93.0% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-28T13:08:28.633 INFO:teuthology.orchestra.run.smithi191.stderr: 92.9% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-28T13:08:28.635 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-04-28T13:08:28.635 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m1.272s 2024-04-28T13:08:28.635 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m4.121s 2024-04-28T13:08:28.635 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.142s 2024-04-28T13:08:28.650 INFO:teuthology.orchestra.run.smithi053.stderr: 93.1% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-28T13:08:28.680 INFO:teuthology.orchestra.run.smithi077.stderr: 92.9% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-28T13:08:28.682 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-28T13:08:28.682 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m1.318s 2024-04-28T13:08:28.682 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m4.721s 2024-04-28T13:08:28.682 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.162s 2024-04-28T13:08:28.683 INFO:teuthology.orchestra.run.smithi053.stderr: 93.0% -- replaced with /var/log/ceph/10525726-055c-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-28T13:08:28.684 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-28T13:08:28.684 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m1.323s 2024-04-28T13:08:28.684 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m5.329s 2024-04-28T13:08:28.684 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.205s 2024-04-28T13:08:28.685 INFO:tasks.cephadm:Archiving logs... 2024-04-28T13:08:28.685 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi053/log 2024-04-28T13:08:28.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T13:08:29.473 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi077/log 2024-04-28T13:08:29.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T13:08:30.162 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi191/log 2024-04-28T13:08:30.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T13:08:30.778 INFO:tasks.cephadm:Removing cluster... 2024-04-28T13:08:30.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10525726-055c-11ef-bc93-c7b262605968 --force 2024-04-28T13:08:30.979 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T13:08:31.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10525726-055c-11ef-bc93-c7b262605968 --force 2024-04-28T13:08:31.455 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T13:08:31.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10525726-055c-11ef-bc93-c7b262605968 --force 2024-04-28T13:08:32.176 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 10525726-055c-11ef-bc93-c7b262605968 2024-04-28T13:08:32.429 INFO:tasks.cephadm:Removing cephadm ... 2024-04-28T13:08:32.429 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T13:08:32.448 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T13:08:32.464 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T13:08:32.479 INFO:tasks.cephadm:Teardown complete 2024-04-28T13:08:32.480 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-28T13:08:32.490 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-28T13:08:32.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T13:08:32.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T13:08:32.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T13:08:32.618 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-28T13:08:32.627 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-28T13:08:32.628 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:08:32.630 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:08:32.633 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:08:32.644 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:08:32.645 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:08:32.646 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:08:32.650 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:08:32.650 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-28T13:08:32.650 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 +1877us[+1820us] +/- 51ms 2024-04-28T13:08:32.650 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -1050us[-1106us] +/- 69ms 2024-04-28T13:08:32.650 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 -1563us[-1563us] +/- 63ms 2024-04-28T13:08:32.651 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:08:32.654 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:08:32.655 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-28T13:08:32.655 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv01.front.sepia.ceph.com 3 6 377 52 +1903us[+1840us] +/- 51ms 2024-04-28T13:08:32.655 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -1013us[-1013us] +/- 69ms 2024-04-28T13:08:32.655 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -1536us[-1536us] +/- 63ms 2024-04-28T13:08:32.655 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:08:32.655 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:08:32.656 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-04-28T13:08:32.656 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 +1895us[+1866us] +/- 51ms 2024-04-28T13:08:32.656 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -1040us[-1040us] +/- 69ms 2024-04-28T13:08:32.656 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -1557us[-1557us] +/- 63ms 2024-04-28T13:08:32.656 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:08:32.656 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-28T13:08:32.666 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-28T13:08:32.666 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-28T13:08:32.675 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T13:08:32.775 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T13:08:32.839 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T13:08:32.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:08:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:08:32.932 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:08:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:08:32.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:08:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:08:32.985 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-28T13:08:32.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-28T13:08:33.005 INFO:teuthology.task.internal:Duration was 2226.425297 seconds 2024-04-28T13:08:33.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-28T13:08:33.014 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-28T13:08:33.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T13:08:33.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T13:08:33.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T13:08:33.053 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:08:33.055 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:08:33.062 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:08:33.498 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-28T13:08:33.498 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-04-28T13:08:33.498 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T13:08:33.520 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-04-28T13:08:33.520 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T13:08:33.563 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-04-28T13:08:33.563 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T13:08:33.588 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-28T13:08:33.588 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T13:08:33.590 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T13:08:33.606 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T13:08:33.654 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-28T13:08:33.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T13:08:33.660 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T13:08:33.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T13:08:34.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-28T13:08:34.037 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-28T13:08:34.037 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T13:08:34.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T13:08:34.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T13:08:34.110 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-28T13:08:34.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:08:34.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:08:34.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:08:34.150 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-04-28T13:08:34.153 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-04-28T13:08:34.172 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-04-28T13:08:34.194 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:08:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:08:34.218 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:08:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:08:34.249 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:08:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:08:34.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-28T13:08:34.274 INFO:teuthology.task.internal:Transferring archived files... 2024-04-28T13:08:34.274 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi053 2024-04-28T13:08:34.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T13:08:34.316 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi077 2024-04-28T13:08:34.316 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T13:08:34.350 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890/remote/smithi191 2024-04-28T13:08:34.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T13:08:34.384 INFO:teuthology.task.internal:Removing archive directory... 2024-04-28T13:08:34.384 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T13:08:34.387 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T13:08:34.389 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T13:08:34.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-28T13:08:34.444 INFO:teuthology.task.internal:Not uploading archives. 2024-04-28T13:08:34.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-28T13:08:34.454 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-28T13:08:34.455 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T13:08:34.457 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T13:08:34.460 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T13:08:34.469 INFO:teuthology.orchestra.run.smithi053.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 13:08 /home/ubuntu/cephtest 2024-04-28T13:08:34.471 INFO:teuthology.orchestra.run.smithi077.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 13:08 /home/ubuntu/cephtest 2024-04-28T13:08:34.487 INFO:teuthology.orchestra.run.smithi191.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 13:08 /home/ubuntu/cephtest 2024-04-28T13:08:34.488 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-28T13:08:34.501 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-28T13:08:34.523 INFO:teuthology.nuke:Checking targets against current locks 2024-04-28T13:08:34.552 DEBUG:teuthology.nuke:shortname: smithi053 2024-04-28T13:08:34.552 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T13:08:34.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:25:11.300093', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMIgahWvWS44IS4b36QLV2QKG/hhVZLE+IK99bTQHlWdD6dTNYCzOaT6U1VxDQZ/pgyYoLGY8ZRRR1T/bVCjObw='} 2024-04-28T13:08:34.580 DEBUG:teuthology.nuke:shortname: smithi077 2024-04-28T13:08:34.580 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T13:08:34.606 DEBUG:teuthology.nuke:shortname: smithi191 2024-04-28T13:08:34.606 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T13:08:34.611 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:25:11.301400', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIAnM+YPplhW31N9E5+qTneFES7GFimTewRdhA/tPH8YZtEhWNilS+lv2ot4eO5COxAZ/K5HCOsX+3PPR/ZdnUU='} 2024-04-28T13:08:34.632 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676890', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:25:11.302753', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK1BN7AJyX3cYGmZJiBPTv1CHSvO8KDjNvOpzKLBm1uX67XwBql0ZibtywniMbW0yxUiWjAlJszo/XP5rh0tuic='} 2024-04-28T13:08:34.641 INFO:teuthology.orchestra.console.smithi053:Power off 2024-04-28T13:08:34.642 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T13:08:34.667 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-04-28T13:08:34.667 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:34.683 INFO:teuthology.orchestra.console.smithi077:Power off 2024-04-28T13:08:34.683 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T13:08:34.695 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-04-28T13:08:34.711 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-04-28T13:08:34.711 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:34.727 INFO:teuthology.orchestra.console.smithi191:Power off 2024-04-28T13:08:34.727 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T13:08:34.741 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-04-28T13:08:34.751 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-04-28T13:08:34.751 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:34.779 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-04-28T13:08:38.696 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:38.742 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:38.781 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:38.808 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-04-28T13:08:38.852 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-04-28T13:08:38.889 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-04-28T13:08:42.812 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:42.853 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:42.890 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:08:42.924 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-04-28T13:08:42.963 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-04-28T13:08:42.999 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-04-28T13:08:43.025 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-04-28T13:08:43.064 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-04-28T13:08:43.100 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-04-28T13:08:43.206 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} duration: 2226.4252965450287 failure_reason: 'Command failed on smithi053 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=65dad90460f445ddb89f664dd60edbc9 status: fail success: false 2024-04-28T13:08:43.206 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-28T13:08:43.298 INFO:teuthology.run:FAIL